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Abstract 

The  United  States  Department  of  Defense  has  been  caught  in  a  continual  cycle  of 
acquisition  reform  since  its  inception  over  two  hundred  years  ago.  The  goal  of 
acquisition  reform  has  always  been  to  find  the  perfect  balance  between  the  amount  of 
program  oversight  and  the  amount  of  flexibility  in  which  program  managers  are  allowed 
to  manage  their  programs.  The  only  truth  learned  throughout  this  process  is  that  defense 
acquisition  does  need  oversight  and  that  there  is  no  cookie  cutter  pattern  for  oversight 
that  will  fit  all  types  of  acquisition  programs  equally  well.  That  being  said,  the  focus  of 
this  thesis  will  be  to  explore  the  foundations  of  oversight  for  programs  following 
Department  of  Defense  Directive  5000-the  defense  acquisition  bible  and  employ  Delphi 
survey  techniques  to  then  develop  an  estimate  for  the  actual  cost  of  oversight  for  defense 
acquisition  programs  that  are  under  the  guidance  of  the  DoDD  5000. 

The  real  value  in  this  research  will  then  be  to  compare  the  oversight  cost  estimate 
for  programs  under  the  DoDD  5000  to  oversight  cost  estimates  developed  using  the  exact 
same  methodology  but  examining  programs  with  different  types  of  oversight. 
Specifically,  space  acquisition  and  communications  acquisition  have  been  operating 
under  a  different  oversight  format  over  the  last  few  years  and  the  interest  is  in 
determining  if  the  changes  have  made  defense  acquisition  any  more  efficient  and  any  less 
costly. 
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A  COMPARATIVE  ANALYSIS  OF  THE  COST  OF  OVERSIGHT 


OF  MAJOR  DEFENSE  ACQUISITION  PROGRAMS  STRICTLY 
UNDER  THE  DIRECTION  OF  THE  DEPARTMENT  OF  DEFENSE 
5000  SERIES  OF  INSTRUCTIONS 

1,0  Introduction 


1.1  Overview 

As  a  nation  we  must  continue  to  reform  our  defense  acquisition  process,  so  we 
can  respond  to  and  defeat  the  unpredictable  threats  of  a  post-Cold  War  world.  On  July 
20,  1995,  then  Under  Secretary  of  Defense  for  Acquisition  and  Technology,  Dr  Paul 
Kaminski,  hosted  a  conference  on  integrated  product  teams  and  in  his  speech  summed  up 
the  importance  of  acquisition  reform  (referring  specifically  to  the  need  for  flexibility  in 
the  5000  series),  “We  must  tailor  not  only  the  acquisition  strategy,  but  the  acquisition 
approval  process  to  the  specific  circumstance  of  individual  programs”  (21:3).  In  his 
speech.  Dr  Kaminski  addressed  the  importance  of  reform  and  flexibility  in  our  defense 
acquisition  approval,  or  oversight,  process  and  we  are  beginning  to  see  his  vision 
implemented  into  today’s  defense  acquisition  process.  It  is  the  defense  acquisition 
approval  process  that  will  be  the  focus  of  this  research  effort.  Specifically,  the  aim  of  this 
research  focuses  on  the  costs  associated  with  the  defense  acquisition  oversight  process. 

1.2  Background 

A  discussion  of  acquisition  oversight  costs  would  be  impossible  without 
discussing  acquisition  reform.  Acquisition  reform  in  and  of  itself  is  a  very  broad,  wide- 
ranging  topic.  In  his  book.  Arming  the  Eagle:  A  History  ofU.S.  Weapons  Acquisition 
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since  1775,  author  Wilbur  D.  Jones,  Jr.  clearly  illustrates  how  the  U.S.  defense 
acquisition  process  was  actually  created  alongside  the  creation  of  our  nation.  He  goes  on 
to  show  how  the  acquisition  process  has  evolved  for  over  two  centuries.  This  research 
will  not  address  as  broad  a  topic  as  the  history  of  defense  acquisition  reform,  but  will 
focus  more  specifically  with  the  reform  of  the  Department  of  Defense  (DoD)  and  its 
governing  body  of  regulations  for  acquisition,  the  DoD  5000  series.  This  exploration  of 
acquisition  reform  will  begin  with  the  development  of  the  DoD  5000  series  in  the  early 
1970s  and  follow  through  to  the  current  defense  acquisition  process.  The  historical 
development  of  the  DoD  5000  series  will  be  covered  in  depth  in  Chapter  2.  The  purpose 
of  this  exploration  of  the  reform  of  the  DoD  5000  series  is  to  support  the  discussion  of 
how  the  DoD  acquisition  system  works  for  those  programs  that  must  strictly  follow  the 
procedures  outlined  in  the  DoD  5000  series.  The  discussion  of  how  the  defense 
acquisition  process  operates  serves  two  purposes;  1)  it  would  be  incomplete  to  discuss 
how  the  acquisition  process  works  today  without  a  discussion  of  how  we  got  there,  and  2) 
that  illustration  will  provide  the  framework  on  which  the  cost  of  oversight  will  be 
evaluated. 

1,2.1  Thinking  Outside  the  Box 

The  reform  of  the  DoD  5000  series  and  how  that  reform  has  affected  the  defense 
acquisition  approval  process,  and  thus  oversight  costs,  would  alone  provide  sufficient 
material  for  an  interesting  research  project.  However,  that  was  not  the  only  type  of 
reform  taken  into  consideration.  Both  the  defense  space  and  missile  acquisition 
processes  have  taken  acquisition  reform  a  step  further  in  essentially  creating  their  own  set 
of  rules.  In  a  memorandum  from  the  Secretary  of  Defense  dated  2  January  2002,  the 
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Ballistic  Missile  Defense  Organization  (BMDO)  was  re-designated  the  Missile  Defense 
Ageney  (MDA)  and  given  mueh  greater  flexibility  in  regards  to  the  aequisition  proeess  as 
stated: 


The  speeial  nature  of  missile  defense  development,  operations,  and  support  ealls 
for  non-standard  approaehes  to  both  aequisition  and  requirements  generation. 

The  memo  went  on  to  outline  a  major  ‘non-standard’  approaeh.  To  eneourage 
flexible  aequisition  praetiee,  I  delegate  to  the  Direetor,  MDA,  authority  to  use 
transaetions  other  than  eontraets,  grants,  and  eooperative  agreements  to  earry  out 
basie,  applied,  and  advaneed  researeh.  The  memo  further  stated  as  part  of  the 
eommitment  to  ensure  flexibility  I  will  support  additional  or  revised  statutory 
authority  as  identified  by  the  Direetor,  MDA,  to  reduee  development  time  and 
enhanee  program  sueeess  (35:2,  4). 

Spaee  aequisition  also  took  a  similar  direetion  when  on  4  Mareh  2002,  in  a 
memorandum  from  the  Under  Seeretary  of  Defense  for  Aequisition  and  Logisties,  the 
Milestone  Deeision  Authority  (MDA-the  final  level  of  hierarehy  in  the  defense 
aequisition  approval  proeess)  was  delegated  to  the  Seeretary  of  the  Air  Foree  for  all  DoD 
Spaee  Major  Defense  Aequisition  Programs.  It  outlined  the  following  in  regards  to 
flexibility  in  dealing  with  the  regulatory  requirements  of  the  DoD  5000  series: 


The  seeretary  of  the  Air  Foree  in  eoordination  with  the  Seeretaries  of  the  Army 
and  the  Navy,  may  implement  further  aetions  with  regard  to  spaee  aequisition 
streamlining.  For  Spaee  MDAP’s,  the  MDA  is  authorized  to  approve  or  waive 
any  exeeptions  to  the  provisions  of  DoD  instruetions  andpublieations...  (1:1) 

The  Seeretary  of  the  Air  Foree  then  re-delegated  MDA  to  the  Underseeretary  of  the  Air 

Foree  in  a  Memo  dated  14  Mareh  2002  (34:1).  On  20  Mareh  2003,  the  Underseeretary  of 

the  Air  Foree  issued  a  memo  in  whieh  he  granted  “an  exemption  and  waiver  to  the 

proeesses  and  proeedures  deseribed  in  DoDI  5000.2  (Aequisition  proeedure  for  Major 

Defense  Aequisition  Programs-MDAPs)  and  related  guidanee  for  all  eurrent  AFPEO/SP 
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(Air  Force  Program  Executive  Officer/Space)  programs  as  well  as  future  space  programs 
executed  under  the  authority  of  the  AFPEO/SP”  (39:1).  By  Spring  2003,  the  DoD 
acquisition  world  had  two  programs,  missile  and  space,  that  were  not  operating  strictly 
under  the  DoD  5000  series.  Throughout  the  remainder  of  this  research,  these  programs 
will  be  referred  to  as  “outside  the  box”  since  they  are  operating  outside  the  framework  of 
rules  of  the  DoD  5000  series.  In  the  communications  acquisition  world,  another  example 
of  acquisition  reform  can  be  seen.  Communications  acquisition  remains  ‘strictly’  under 
the  procedures  outlined  in  the  DoD  5000  series  yet  those  procedures  are  streamlined  by 
implementing  them  into  a  virtual  world.  For  example,  rather  than  conducting  all 
meetings  face-to-face,  in  a  designated  meeting  location,  the  communications  initiative 
strives  to  save  time  and  money  by  posting  all  necessary  information  in  a  shared  database 
and  then  conducting  meetings  via  the  web.  These  communication  acquisition  programs 
will  hereafter  be  referred  to  as  “virtual  box”  programs  since  they  still  operate  under  the 
framework  of  the  DoD  5000  series,  but  attempt  to  do  so  in  a  virtual,  or  web-based, 
environment.  While  this  research  will  evaluate  the  cost  of  oversight  for  MDAP  programs 
required  to  strictly  follow  the  DoD  5000  series,  as  part  of  a  collaborative  effort,  two 
additional  research  efforts  will  evaluate  the  cost  of  oversight  for  programs  following  the 
more  flexible  space  and  communications  initiatives. 

1.3  Problem 

This  research  effort  will  evaluate  the  cost  of  oversight  of  MDAPs  that  must 
operate  strictly  under  the  guidelines  of  the  DoD’s  governing  acquisition  instructions. 

Why  is  this  study  of  the  cost  of  oversight  of  major  defense  acquisition  programs 
important?  Acquisition  programs  produce  the  technologically  advanced  weaponry  that 
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warfighters  need  to  defend  the  nation  and  the  many  other  nations  who  depend  on  the 
United  States.  We  must  eontinue  to  reform  our  aequisition  proeess  with  the  goal  of 
redueing  oversight  eosts  and  producing  more  advanced  technological  products  and  faster 
in  order  to  be  prepared  for  whatever  threats  may  come.  In  his  testimony  before  the 
Senate  Armed  Services  Committee  1 1  January  2001,  the  Secretary  of  Defense  addressed 
the  issue  of  reforming  the  acquisition  process  and  its  importance  to  the  defense  of  the 
nation; 

The  legacy  of  obsolete  institution  structures  and  processes  and  organization  does 
not  merely  create  unnecessary  cost,  which  of  course  it  does;  it  also  imposes  an 
unacceptable  burden  on  national  defense,”  he  said,  “In  certain  respects,  it  could 
be  said  that  we  are  in  a  sense  disarming  or  ‘under  arming’  by  our  failure  to  reform 
the  acquisition  process  and  to  shed  unneeded  organization  and  facilities  (23:4). 

The  Secretary’s  testimony  put  the  importance  of  acquisition  reform  into  clear 

terms.  He  emphasized  that  we  are  risking  the  nation’s  defense  if  we  do  not  continue  to 

reform  the  way  our  country  arms  itself.  The  final  evidence  in  support  of  the  importance 

of  studying  the  cost  of  oversight  and  acquisition  reform  in  general  comes  in  a 

memorandum  dated  5  January  2001  from  the  then  Undersecretary  of  Defense  for 

Acquisition  and  Technology,  in  his  endorsement  to  the  assistant  service  secretaries  for 

the  establishment  of  a  project  to  create  a  history  of  defense  acquisition  and  confirmation 

of  the  services’  agreement  to  support  the  effort  by  funding  $250,000  per  year  for  fiscal 

years  2002-2006.  He  asserts  the  importance  of  this  project: 


During  the  more  than  fifty  years  since  the  National  Security  Act  of  1947,  the 
Department  of  Defense  acquisition  function  has  experienced  great  change  and 
received  extraordinarily  high  public  visibility  and  congressional  attention.  We  are 
missing  however  a  comprehensive  record  of  Defense  Acquisition 
accomplishments  and  failures  from  which  we  may  have  opportunity  to  learn.  An 
official  history  of  the  Department  of  Defense  Acquisition  System  would  clearly 
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fill  this  historical  void  and  serve  as  a  referenee  and  instruetional  tool  for  the  entire 
aequisition  community,  including  our  educational  institutions  (22:1). 

1.4  Scope  and  Definitions 

What  is  meant  by  “oversight”?  The  term  “oversight”  has  several  meanings  in 
regard  to  MDAPs.  In  order  to  narrow  the  seope,  it  is  helpful  to  first  state  two  types  of 
oversight  that  this  researeh  effort  will  not  address.  These  include  congressional  oversight 
of  the  defense  aequisition  proeess  and  the  oversight  of  the  defense  industrial  base 
(contraetors).  This  researeh  will  not  investigate  either  of  these  eases.  For  the  purpose  of 
this  researeh,  oversight  is  defined  as  the  vertieal  levels  of  approval,  or  the  hierarehy  of 
approval  stages,  that  a  program  must  pass  through  in  order  to  advance  from  one 
aequisition  lifeeycle  stage  (milestone)  to  the  next.  This  researeh  captures  the  costs  that 
are  generated  as  the  program  is  reviewed  and  approved  at  eaeh  level  in  the  vertieal 
hierarehy  as  it  moves  toward  final  approval  from  the  governing  aequisition  board  through 
the  milestone  approval  proeess.  This  researeh  effort  will  study  major  defense  aequisition 
programs  in  the  horizontal  timeline  between  Milestone  B  (program  initiation)  through  the 
Low  Rate  Initial  Produetion  (TRIP)  approval  at  Milestone  C,  and  will  look  at  the  vertical 
approval  hierarchy  from  above  the  Program  Manger’s  (PM)  level  up  to  the  Defense 
Acquisition  Executive  (DAE)  or  Component  Acquisition  Executive  (CAE)  at  the  Defense 
Acquisition  Board  (DAB). 

Another  set  of  terms  from  the  statement  of  the  problem  that  needs  explanation  is 
‘operating  strietly  under  the  DoD  5000  series  of  instruetions.’  Throughout  the  remainder 
of  this  researeh,  programs  that  must  strictly  follow  the  rules  outlined  in  the  DoD  5000 
series  will  be  referred  to  as  “box”  programs;  for  they  must  operate  inside  the  “box”,  or 
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framework,  of  rules  outlined  in  the  DoD  5000  series.  The  assumption  is  that  sinee  the 
DoD  5000  series  governs  defense  aequisition  programs,  all  DoD  aequisition  programs 
would  have  to  operate  strietly  under  this  regulation.  However,  as  mentioned  previously, 
both  the  spaee  and  missile  aequisition  proeesses  have  been  granted  authority  to  operate 
“outside  the  box”  environment  so  that  is  not  the  ease.  These,  along  with  other  key  terms, 
will  be  defined  in  greater  detail  in  Chapter  2. 

1.5  Research  Objectives  and  Questions 

The  objeetive  of  this  researeh  effort  is  to  evaluate  the  eost  of  oversight  of  “box” 
MDAPs.  Until  reeently,  all  aequisition  programs  were  “box”  programs.  Reeent 
aequisition  reforms  in  both  the  eommunieations  and  spaee  and  missile  aequisition 
environments,  however,  are  attempting  to  make  their  aequisition  proeesses  more  effieient 
by  either  “virtualizing”  box  requirements,  or  developing  their  own  set  of  aequisition 
rules.  The  measuring  stiek  for  how  effeetive  these  reforms  are  is  how  their  oversight 
eosts  eompare  to  eaeh  other  and  to  programs  still  operating  “in  the  box”.  The  final 
question  to  be  answered  by  this  researeh  effort  is,  “What  are  the  key  oversight  eost 
drivers  for  “box”  MDAPs?”  Determining  eost  of  oversight  alone  is  helpful,  but  it  is  more 
benefieial  to  determine  the  faetors  that  drive  those  eosts  of  oversight  in  order  to  better 
foeus  efforts  to  make  DoD  aequisition  more  effieient. 
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1.6  Summary 

This  chapter  outlined  the  importance  of  this  researeh  effort,  some  initiatives  that 
drove  this  groundbreaking  researeh  effort,  and  the  importanee  of  studying  the  evolution 
of  the  DoD  5000  series.  Chapter  2  will  further  discuss  other  researeh  and  studies  that 
have  been  condueted  dealing  with  the  cost  of  oversight  of  MDAPs  as  well  as  defining 
key  terms  and  solidifying  the  scope  of  the  researeh  effort.  Chapter  3  will  fill  in  the 
blanks  left  by  Chapter  2’s  discussion  of  how  the  oversight  proeess  works  by  employing 
the  Delphi  Method  to  determine  an  estimate  for  the  eost  of  the  oversight  proeess.  The 
primary  aim  of  Chapter  3  is  to  colleet  the  data  neeessary  to  answer  the  research  question, 
“What  is  the  cost  of  oversight  of  major  acquisition  programs  operating  inside  the  box?” 
An  interesting  perspeetive  ean  be  gained  by  then  viewing  that  cost  of  oversight  as  a 
percentage  of  overall  program  cost  and  then  eomparing  this  percentage  to  the  percentages 
for  the  eommunieations  aequisition  programs  that  are  operating  in  a  more  flexible  box 
and  the  spaee  and  missile  programs  that  are  operating  outside  the  box.  Chapter  4  will 
analyze  the  data  colleeted  using  Chapter  3  procedures  and  provide  conclusions.  Chapter 
5  will  diseuss  further  researeh  potential  in  the  area  of  evaluating  the  cost  of  oversight. 
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2,0  Literature  Review 


2,1  Overview 

The  cost  of  oversight  of  Major  Defense  Acquisition  Programs  (MDAP) 
significantly  impacts  the  overall  Department  of  Defense  (DoD)  budget.  Hence,  of  the 
hundreds  of  billions  of  dollars  the  United  States  spends  every  year  in  supporting  its 
defense  programs,  a  portion  of  that  budget  is  applied  to  the  oversight  of  those  programs. 
According  to  one  study,  however,  “The  Department  of  Defense  loses  approximately  $5 
billion  per  year  in  investment  program  content  due  to  cost  growth”  (9:60).  This  study 
shows  that  despite  DoD’s  attempt  at  sound  oversight  of  its  acquisition  programs,  those 
programs  continue  to  be  hampered  by  cost  growth.  The  DoD  continues  to  evolve  its 
oversight  philosophy  through  policy  in  an  attempt  to  find  the  best  balance  between 
control  and  flexibility.  They  do  this  because  if  our  nation  stops  looking  at  the  cost  of  the 
oversight  of  our  defense  programs,  program  costs  could  continue  to  escalate  until  the 
acquisition  system  is  no  longer  able  to  produce  the  quality  products  our  war  fighters  need 
to  defend  this  nation. 

The  first  section  of  this  chapter  describes  how  a  typical  MDAP  operates  as 
outlined  in  the  DoD  5000  series  of  instructions.  Key  terms  and  definitions  are  provided 
as  well  as  review  of  some  of  those  initially  mentioned  in  Chapter  1 .  Next,  to  provide 
background  for  the  operation  of  today’s  acquisition  system,  the  evolution  of  the  DoD 
5000  series  of  instructions  is  explored.  The  final  section  of  this  chapter  discusses 
previous  research  in  the  area  of  the  cost  of  oversight  of  MDAPs.  The  major  issue,  stated 
in  Chapter  1  concerns  the  cost  of  the  vertical  levels  of  approval  (oversight)  in  the  MDAP 
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process  that  enables  an  MDAP  to  move  from  one  development  stage,  or  milestone,  to  the 
next. 

2.2  MDAP  Operation  as  Defined  in  the  DoD  5000  Instructions 

The  MDAP  proeess  is  governed  by  the  DoD  5000  series  of  instructions.  Since  its 
ereation  in  1970,  the  DoD  5000  series  has  been  changed  several  times;  with  the  most 
reeent  version  dated  12  May  2003.  This  seetion  explains  the  rules  established  by  the 
eurrent  instruetions  and  how  the  MDAP  proeess  is  designed  to  work.  The 
aforementioned  deseriptions  provide  a  framework  of  the  regulatory  environment  in  whieh 
an  MDAP  must  operate.  This  researeh  will  evaluate  the  eost  of  that  oversight  of 
MDAP’s  based  on  the  rules  outlined  in  the  DoD  5000  series  for  those  programs  that  must 
strietly  follow  these  instruetions. 

2.2.1  Deflnitions 

Some  key  terms  need  to  be  defined  prior  to  establishing  the  regulatory  framework 
under  which  a  box  program  must  operate.  To  begin,  a  MDAP  is  defined  based  on  dollar 
thresholds.  As  outlined  in  enelosure  2  of  DoDI  5000.2,  a  Major  Defense  Aequisition 
Program  (MDAP)  (also  referred  to  as  Aequisition  Category  (ACAT)  I)  is,  “estimated  by 
the  Under  Seeretary  of  Defense  for  Aequisition,  Teehnology,  and  Logisties  to  require 
research,  development,  test,  and  evaluation  (RDT&E)  of  more  than  $365  million  in  fiseal 
year  2000  eonstant  dollars  or,  for  proeurement  of  more  than  $2,190  billion  in  fiseal  year 
2000  eonstant  dollars”  (15:16). 

Oversight  refers  to  the  vertieal  levels  of  approval  above  the  PM.  Figure  2.1 
shows  the  vertieal  levels  of  approval,  or  oversight,  above  the  PM’s  level. 
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Figure  2.1;  Oversight  Approval  Levels 


According  to  Figure  2.1,  the  vertical  levels  of  hierarchy  appear  to  be  quite  streamlined.  It 
is  important  to  note  that  although  the  figure  appears  to  put  the  OIPT  and  the  WIPT  on  the 
same  level  of  the  hierarchy,  that  in  matters  of  practice,  the  WIPT  actually  falls  under  the 
OIPT.  The  analysis  discussed  in  chapter  4  dissects  these  levels  in  order  to  determine  how 
many  meetings  it  takes  before  a  program  is  ready  to  go  to  the  DAB  to  be  reviewed  for 
milestone  approval,  and  how  many  people  at  which  pay  grades  are  involved.  The  PM  is 
responsible  for  the  efficient  operation  of  the  program  he  is  charged  with  executing.  PM’s 
are  responsible  to  the  next  level  in  the  hierarchy,  the  Program  Executive  Officer  (PEO). 
According  to  Enclosure  9  of  the  5000.2,  a  PEO  is  assigned  to  all  MDAP’s  unless  a 
waiver  is  granted  (15:35).  The  PEO’s  sole  responsibility  is  executive  management  over 
the  PM  or  PM’s  to  which  they  are  assigned.  The  final  level  in  the  formal  vertical  chain 
of  command  is  the  MDA  who  chairs  the  DAB.  The  MDA  will  be  the  DAE  or  CAE 
(depending  on  whether  the  program  is  under  the  authority  of  the  DoD  or  Service 
Component,  respectively).  The  MDA  is  the  final  approval  authority  in  determining 
whether  a  program  moves  horizontally  from  one  milestone  to  the  next.  Outside  the 
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formal  vertical  oversight  chain,  but  still  critical  to  the  oversight  process,  are  the 
Integrated  Process 

Teams  (both  the  OIPT-Overarching  and  WIPT-Working  Level).  According  to  the  DODI 
5000.2,  “An  OIPT  shall  facilitate  program  communications  and  issue  resolution,  and 
support  the  MDA  for  ACAT  1  and  lA  (‘ACAT  I’  refers  to  MDAP  and  ‘A’  refers  to 
communications)  programs”  (15:12).  An  Integrated  Product  Team  (IPT)  is  a  group  of 
functional  experts  with  a  stake  in  the  operation  and  success  of  a  program.  The  IPT 
establishment  and  use  of  IPTs  were  addressed  in  a  1995  from  then  Under  Secretary  of 
Defense  for  Acquisition  and  Technology,  Dr.  Paul  Kaminski.  In  his  memo,  he  stated  that 
the  purpose  of  IPTs  would  be  to  provide  for  early  and  on-going  oversight  of  defense 
acquisition  programs  rather  than  oversight  only  at  the  six-month  out  point  from  MDA 
review.  IPTs  were  officially  codified  in  the  15  March  1996  version  of  the  DoD  5000 
series  (18:48). 

2,2,2  Development  Stages 

Each  MDAP  moves  through  horizontal  steps  that  represent  movement  from  one 
development  stage  to  the  next.  Figure  2.2  shows  the  horizontal  steps  that  a  program  must 
progress  through. 


Process  entry  at  Milestones  A,  B,  or  C 
Entrance  criteria  met  before  entering  phase 

Evolutionary  Acquisition  or  Single  Step  to  Full 
Capability 


(Program 

Initiation) 


IOC 


FOC 


Concept 

Refinement 

Technoiogy 

Deveiopment 

System  Deveiopment 
&  Demonstration 

Production  & 
Deployment 

Operations  & 
Support 

\  Concept 
/  Decision 

Design 

C  y  Readiness 
Review 

LRIP/IOT&E  ^  Decision 

V  Review 

Pre-Systems  Acquisition 


Systems  Acquisition 


Sustainment 


Figure  2.2:  Forward  Progress  (15:2) 
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The  “systems  aequisition”  process  is  composed  of  all  the  steps  included  within  Milestone 
B  and  Milestone  C.  There  are  steps  under  Milestone  C  that  continue  on  after  a  TRIP 
decision,  but  these  are  outside  the  scope  of  this  research.  “The  purpose  of  the  System 
Development  and  Demonstration  (SDD)  phase  is  to  develop  a  system  or  an  increment, 
capability”  (15:6).  The  entry  point  to  SDD  is  Milestone  B.  While  the  refinement  of  ideas 
and  needs  occurring  in  Milestone  A  may  be  considered  the  conception  of  a  program,  it  is 
not  until  a  program  is  approved  to  enter  Milestone  B  (program  initiation)  that  the 
program  is  established.  The  tables  in  Enclosure  3  of  DoDI  5000.2  list  all  of  the  statutory 
and  regulatory  requirements  that  must  be  met  at  each  milestone  decision  point. 

The  purpose  of  system  development  is,  “the  management  and  mitigation  of 
technology  risk”  as  well  as,  “. .  .objective  assessment  of  technology  maturity...”  (15:8).  In 
order  to  move  from  development  to  demonstration,  the  system  must  undergo  a  Design 
Readiness  Review  (DRR),  which  “provides  an  opportunity  for  mid-phase  assessment  of 
design  maturity”  (15:8).  After  successfully  completing  the  DRR,  a  program  enters  into 
the  demonstration  phase.  The  intent  of  the  system  demonstration  phase  is  to 
“demonstrate  the  ability  of  the  system  to  operate  in  a  useful  way,”  and  the  system  exits 
the  demonstration  phase  when  “a  system  is  demonstrated  in  its  intended  environment, 
using  the  selected  prototype;  meets  approved  requirements;  industrial  capabilities  are 
reasonably  available;  and  the  system  meets  or  exceeds  exit  criteria  and  Milestone 
entrance  requirements”  (15:8).  The  final  result  of  the  demonstration  portion  of  Milestone 
B  is  “dependent  on  a  decision  by  the  MDA  to  commit  the  program  at  Milestone  C  or  a 
decision  to  end  the  effort”  (15:8). 
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A  MDAP  enters  LRIP  when  the  MDA  approves  the  program  to  enter  Milestone 
C.  The  purpose  of  LRIP  is  to  “result  in  completion  of  manufacturing  development  in 
order  to  ensure  adequate  and  efficient  manufacturing  capability  and  to  produce  the 
minimum  quantity  necessary  to  provide  production  or  production  representative  articles 
for  . .  .<testing>”  (15:9).  LRIP  is  considered  Tow  rate’  because  the  production  quantities 
are  limited  in  order  to  provide  enough  units  to  ensure  testing  and  producibility  yet  limit 
the  taxpayer  expense  before  fully  approving  production.  According  to  DoDI  5000.2,  the 
quantity  is  normally  limited  to,  “10  percent  of  the  total  production  quantity  documented 
in  the  acquisition  strategy”  (15:9). 

2.3  Evolution  of  the  DoD  5000  Series  of  Instructions 

With  the  environment  in  which  a  “box”  MDAP  must  operate  described,  the  next 
step  is  to  discuss  the  evolution  of  the  DoD  5000  series.  In  his  book.  Arming  the  Eagle, 
Wilbur  D.  Jones,  Jr.  refers  to  a  quotation  by  Robert  T.  Marsh  that  appeared  in  “U.S. 
Defense  Policy  in  an  Era  of  Constrained  Resources”  that  is  appropriate  to  the  exploration 
of  the  evolution  of  defense  acquisition’s  5000  series  of  instructions,  “Every 
administration  and  Congress  since  [WWII]  has  instituted  changes  to  improve  the 
[acquisition]  organization  and  process,”  Marsh  continued,  “As  one  might  expect,  these 
changes  did  not  always  bring  the  improvement  desired,  and  in  fact  sometimes  created 
new  problems,  more  serious  than  the  ones  for  which  the  cures  were  intended”  (27:400). 
The  DoD  5000.1  wasn’t  conceived  until  1970  (27:408). 
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2.3.1  Importance  of  studying  the  DoD  5000’s  Development 

According  to  an  article  by  Joe  Ferrara  on  the  evolution  of  the  DoD  5000  series, 
DoDI  5000.1  and  its  accompanying  DoDI  5000.2,  “have  been  the  foundation  of  the 
defense  acquisition  process  for  over  20  years”  (20:109).  Ferrara  points  out  that  from 
1971  to  1993,  the  DoD  5000  series  was  reissued  nine  different  times  (20:109).  A 
revision  in  1996  and  others  in  2000  (2002  for  the  5000.2)  and  2003  (for  both  the  5000.1 
and5000.2)  brings  to  total  of  12  the  times  that  this  governing  set  of  documents  has  been 
rewritten.  Ferrara  points  out  that  the  reason  it  is  important  to  study  the  evolution  of  the 
DoD  5000  series  is  that,  “the  5000  documents  offer  a  unique  window  on  the  evolution  of 
policy  in  a  major  government  department”  (20:109).  The  DoD  5000  series  is  at  the  heart 
of  aequisition  reform  efforts  as  the  tool  that  each  administration  has  used  to  implement 
their  vision  of  how  to  streamline  the  defense  aequisition  system. 

2.3.2  Secretary  Packard  Leads  the  Way 

The  5000  series  came  about  under  the  direetion  of  David  Packard,  President 
Nixon’s  Deputy  Seeretary  of  Defense.  Deputy  Seeretary  Packard  headed  a  defense 
aequisition  review  council  charged  to  examine  the  defense  aequisition  proeess  to  discover 
opportunities  to  improve  the  proeess.  In  May  1970,  Packard  issued  a  memorandum  in 
whieh  the  DoD  5000  series  was  conceived  (20: 111).  This  memorandum  outlined  the 
ideas  that  would  later  form  the  basis  for  the  first  issuance  of  DoD  Directive  5000.1. 

Some  of  Packard’s  ideas  listed  included,  “deeentralized  exeeution,  streamlined 
management  structures,  and  use  of  appropriate  contraet  meehanisms”  (20:1 1 1). 

Packard’s  ideas  beeame  the  central  themes  throughout  the  first  DoDI  5000.1,  issued  in 
July  1971  (20:1 1 1).  Ferrara  suggests  the  original  guidelines  for  the  operation  of  a 
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defense  aequisition  program  as  outlined  in  the  5000.1  (as  envisioned  by  Deputy  Seeretary 
Paekard)  have  been  the  driving  foree  behind  every  aequisition  reform  effort  and  DoD 
5000  revision  ever  since  (20; 111).  A  key  point  excerpted  from  the  first  DoD  5000 
document  was  that  “Layers  of  authority  between  the  program  manager  and  his 
Component  Head  shall  be  kept  to  a  minimum”  (20; 111).  Over  three  decades  ago,  Deputy 
Secretary  Packard  knew  the  importance  of  the  removal  of  unnecessary  layers  of 
oversight.  With  publication  of  the  document  that  laid  the  groundwork  for  this  important 
concept,  what  each  revision  since  has  attempted  to  do  is  find  the  balance  of  oversight  that 
provides  the  greatest  amount  of  flexibility  and  ensures  the  most  efficient  deployment  of 
products  to  the  warfighter. 

2,3.2. 1  Packard’s  Key  Concepts 

Reaffirming  his  idea  that  the  concepts  outlined  in  the  first  DoD  5000  Directive 

were  the  basis  for  all  future  iterations  of  the  document,  Ferrara  makes  the  point  that. 

The  founding  5000.1  set  the  tone  and  all  subsequent  documents  have  been 
remarkably  consistent  in  continuing  to  articulate  a  few  key  themes.  This  is 
remarkable  because  as  even  the  most  casual  observer  of  the  DoD  procurement 
scene  is  aware,  the  last  two  decades  have  witnessed  extraordinary  and  persistent 
agitation  for  reform  and  improvement  (20;1 13). 

Ferrara  lists  the  central  themes  consistent  with  all  of  the  DoD  5000  issuances; 
Centralized  policy  decentralized  execution;  fly  before  you  buy;  streamlined  organization; 
limited  reporting  requirements;  and  program  stability  (20;  113-115).  In  brief,  the  first 
theme,  centralized  policy-decentralized  execution  means  bringing  authority  to  execute  the 
program  to  the  lowest  level  possible  while  ensuring  policy  is  stable  and  consistent  for  all 
parties  involved.  Fly  before  you  buy  stresses  the  importance  of  testing.  Taxpayer  dollars 
should  not  be  committed  to  a  program  until  it  first  proves  useful  to  the  warfighter  and 
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producible  given  the  eurrent  industrial  base  and  teehnology.  Streamlined  organizations 
toueh  on  the  importanee  of  the  removal  of  exeessive  layers  of  management,  as  they  are 
detrimental  to  the  effieient  operation  of  the  proeess.  Limited  reporting  requirements 
attempts  to  remove  duplieated  efforts.  Ferrara  ealled  these  themes  the  “management 
prineiples  etehed  in  the  granite  of  the  [first]  5000.1”  (20:1 13)  and  supports  this 
observation  in  several  instanees  by  eomparing  how  different  revisions  of  DoDI  5000.1 
address  and  ineorporate  the  key  themes  in  a  similar  manner. 

2,3,3  Changes  by  Administration 

Ferrara  asserts  that  the  prime  driver  behind  efforts  to  revise  the  DoD  5000  series 
has  been  changes  in  presidential  administration.  Aeeording  to  Ferrara,  the  DoDI  5000.1 
was  first  issued  in  1971  under  Nixon  with  two  revisions  under  President  Ford 
(1975,1977);  one  revision  under  Carter  (1980);  four  revisions  under  Reagan 
(1982,1985,1986,1987);  one  revision  under  Bush(1991)  and  one  revision  under  Clinton 
(1993)  (20:1 15).  There  is  also  a  seeond  reissuance  under  Clinton  (1996)  and  then  two 
revisions  under  Bush  (2000,  2003). 

2,3,3, 1  The  Nixon  Administration  (1968-1974) 

The  action  to  begin  the  DoD  5000  series  was  eoneeived  in  1970  in  response  to 
rising  defense  aequisition  costs  (20:1 10).  The  first  DoD  5000  instruction  outlined  both 
the  vertieal  layers  of  hierarehy  and  the  horizontal  steps  that  a  program  must  move 
through  to  reaeh  full  production.  Under  the  first  series,  the  horizontal  steps  ineluded 
program  initiation,  full-seale  development,  and  produetion/deployment  (20:1 12).  These 
steps  are  somewhat  similar  to  today’s  milestones  A  through  C  steps;  however,  the  vertieal 
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final  approval  authority  for  moving  form  one  milestone  to  the  next  went  all  the  way  to  the 
Seeretary  of  Defense. 

2,3.3.2  The  Ford  Administration  (1974-1977) 

In  1975  the  reissuanee  of  DoDI  5000.1  eame  with  the  issue  of  DoDI  5000.2 
(20;  116).  This  new  doeument  attempted  to  bring  more  eoneentrated  foeus  to  the  series, 
making  the  DoD  5000  series  more  user  friendly. 

The  1977  revision  eame  in  response  to  “the  recommendations  of  the  commission 
on  government,  the  establishment  of  the  Office  of  Federal  Procurement  Policy,  and  the 
issuance  of  Office  of  Management  and  Budget  Circular  A- 109  (20: 117).  The  new  change 
in  the  1977  version  instituted  a  new  milestone  decision  point;  Demonstration  and 
Validation  (20:1 17).  This  initiative  attempted  to  mitigate  technical  risks  as  early  as 
possible  in  the  life  of  a  program.  Ferrara  asserts  that  this  event  was  likely  brought  about 
in  part  due  to  the  large  amount  of  money  being  spent  to  keep  up  with  the  Russians  during 
the  Cold  War  (20:1 17). 

23.3.3.  The  Carter  Administration  (1977-1981) 

In  the  1980  revision,  the  Carter  administration  attempted  to  reduce  cycle  time  in 
order  to  get  products  to  the  warfighter  more  quickly  and  add  more  detail  in  the  form  of 
requiring  new  documents.  In  support  of  reducing  cycle  time,  this  version  authorized 
services  to  do  some  novel  things  including,  “omitting  phases  altogether”  (20:1 18).  To 
add  more  detail,  the  1980  revision  required  a  new  document  known  as  the  Integrated 
Program  Summary  (IPS).  According  to  the  revision,  the  purpose  of  the  IPS  was  to 
provide  a  document  in  which  the  service  in  charge  of  the  program  could  summarize  the 
implementation  plan  for  the  life  cycle  of  the  product  being  developed  (20: 118). 
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2,3,3,4  The  Reagan  Administration  (1981-1989) 

1985’s  revision  was  in  response  to  the  aequisition  horror  stories  about  $900 
hammers  and  $500  toilet  seats.  Jones  Jr.  deseribes  how  these  stories  affected  the  climate 
in  Congress  in  his  book  Arming  the  Eagle,  “Congress  at  mid-decade  was  overloaded  with 
some  150  different  defense  procurement  bills  in  the  hopper,  many  counter  productive  and 
contradictory”  (27:374).  The  1985  version  created  the  DAE  to  act  as  a  single 
accountable  point  of  contact  over  the  approval  of  each  acquisition  program  (20:1 19). 

1986  through  1987  was  a  time  of  great  change  for  the  DoD  5000  series  and  for  the 
defense  acquisition  system  overall.  In  1986,  Congress  enacted  the  Defense  Acquisition 
Improvement  Act  to  implement  the  Packard  Commission  recommendations  (20:120). 

One  major  step  coming  from  the  act  was  the  creation  of  the  Under  Secretary  of  Defense 
for  Acquisition.  The  1987  series  revision  actually  put  a  streamlined  chain  of  command 
into  place  for  the  acquisition  process,  which  ran  from  the  PM  thru  the  PEO  to  the 
Acquisition  Executive.  Previously,  the  role  of  the  acquisition  executive  and 
corresponding  role  of  milestone  decision  authority  were  held  by  the  Secretary  of  Defense. 

Another  bold  move  made  by  the  new  Under  Secretary  of  Defense  for  Acquisition 
was  the  creation  of  committees  to  “provide  assistance  in  program  review  and  policy 
formulation”  (20:120).  There  were  several  committees,  each  specializing  in  a  specific 
function  (i.e.,  science  and  technology  and  nuclear  issues).  The  reason  for  this  was  to 
streamline  and  cut  down  on  the  number  of  committees  that  met  with  the  new  Under 
Secretary  as  the  chair  of  the  DAB  and  MDA.  The  article  states  that  at  “one  count  [the 
number  of  committees]  went  as  high  as  126  separate  boards  and  councils”  (20:120). 
Committees  were  never  fully  adopted  however;  Eerrara  credits  this  pioneering  vision 
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with  establishing  modern  day  OIPTs  (20:120).  These  teams,  working  with  today’s 
defense  aequisition  hierarehy,  are  implementing  the  vision  of  an  overarehing  body  of 
funetional  experts  helping  to  faeilitate  the  aequisition  proeess,  so  that  by  the  time  a 
program  is  brought  before  the  DAB,  all  of  the  problems  are  smoothed  out,  mitigating  the 
need  for  the  hundreds  of  meetings.  This  new  OIPT  and  a  WIPT  were  created  as  part  of 
the  DoD  5000  series  revision  published  on  15  March  1996  (20:120). 

2.3.3. 5  The  Bush  Administration  (1989-1993) 

The  objectives  of  the  1991  revision  were  to  create:  1)  a  uniform  system  of 
acquisition  policy,  2)  provide  rigid  guidelines  for  programs  through  the  acquisition  life 
cycle-did  not  allow  services  to  supplement  the  DoD  5000  series,  3)  made  the  DoDI 
5000.2  applicable  to  all  acquisition  programs  (not  just  MDAPs),  and  4)  provide  that  all 
necessary  information  would  be  transmitted  in  writing  (a  clear  departure  from  Packard’s 
vision  of  less  paperwork).  The  1991  revision  consisted  of  over  900  pages  where  previous 
versions  since  1971  failed  to  exceed  60  pages  (20:122).  The  1991  version  burdened  the 
defense  acquisition  process  by  requiring  paperwork  for  everything  and  actually  removing 
all  flexibility  by  forbidding  any  waivers  to  the  instructions. 

2.3.3. 6  The  Clinton  Administration  (1993-2001) 

Ferrara  makes  little  mention  of  the  1993  revision;  he  gives  special 
attention  to  contrasting  the  policies  that  were  implemented  in  Bush’s  1991  revision  and 
Clinton’s  1996  revision.  Ferrara  states: 

The  1991  documents  represented  a  dramatic  centralization  of  policy  control  and 
procedural  specificity.  And  the  1996  version  represents  an  equally  dramatic 
reversal  of  these  elements  (20:121)! 
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The  1996  revision  was  the  antithesis  of  the  1991  version  as  it  attempted  to  re¬ 
instill  the  Packard  spirit  into  the  regulations.  The  1996  version  reversed  the  decision  to 
make  the  5000.2  applicable  to  all  programs  in  an  attempt  to  give  more  authority  and 
flexibility  to  components  to  run  their  programs  efficiently,  (again,  Packard’s 
decentralized  execution).  The  1996  version  also  attempted  to  respond  to  the  changing 
world  environment  brought  on  since  the  end  of  the  Cold  War.  Since  threats  to  the  United 
States  could  come  from  anywhere  at  anytime,  the  acquisition  system  needed  the 
flexibility  to  be  able  to  respond  very  quickly.  The  1996  version  instituted  the  concept  of 
“Advanced  Concept  Technology  Demonstrations”  in  order  to  infuse  new  technology  into 
the  process  (20:123).  Another  major  break  through  in  the  1996  version  was  the 
institutionalization  of  Integrated  Product  Teams  to,  “breakdown  the  barriers  between 
different  organizations  and  acquisition  disciplines  and  encourage  integrated  solutions  to 
management  problems  (20:123).  Finally,  the  1996  revision  required  less  paperwork  than 
the  1991  version  by  canceling,  “numerous  report  formats  previously  mandated  in  the 
1991  documents”  (20:123). 

2.3.3  J  The  Bush  Administration  (2001-present) 

According  to  an  article  written  for  National  Defense  Online,  president  Bush’s 
Secretary  of  Defense,  Donald  Rumsfeld  demanded  the  “transformation  of  the  Defense 
Department  business  practices,  for  greater  innovation  and  flexibility  in  weapons 
acquisition”  from  the  time  he  stepped  into  office  (19:3).  As  the  bible  for  DoD 
Acquisition,  the  DoD  5000  series  was  naturally  one  of  Rumsfeld’s  prime  vehicles  for 
codifying  his  “transformation”.  In  his  memo  canceling  the  DoD  5000.1  and  5000.2  dated 
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2000  and  2002  respectively,  Rumsfeld’s  Deputy  Secretary  of  Defense  Paul  Wolfowitz 
states: 

I  have  determined  that  the  current  subject  documents  require  revision  to  create 
an  acquisition  policy  environment  that  fosters  efficiency,  flexibility,  creativity  and 
innovation  (44:1). 

According  to  the  National  Defense  article,  the  reason  the  defense  department 
sought  to  again  revise  the  DoD  5000  series  in  2003  was  that  previous  attempts  at 
instilling  flexibility  in  the  regulations,  .  .have  not  gone  far  enough  because  they  have 
not  addressed  adequately  the  need  for  more  innovation  and  efficiency”  (19:1).  The 
author  of  the  article  asserts  that  senior  defense  officials  are  still  frustrated  because, 

“. .  .many  weapons  programs  are  years  behind  schedule,  as  a  result  of  a  cumbersome 
procurement  process,  and  that  acquisition  managers  don’t  work  as  efficiently  as 
commercial  businesses  do,  because  they  are  restricted  by  the  rules”  (19:2).  The  author 
theorizes  that  despite  all  of  the  previous  revisions  to  the  DoD  5000  series,  defense 
officials  see  the  instructions  as  requiring  too  much  oversight  and  that  the  oversight  is 
slowing  down  the  process. 

In  his  briefing  entitled  “Evolutionary  Acquisition  Update  and  the  DoD  5000 
Revision,  Skip  Hawthorne  summed  up  the  pitfalls  of  the  DoD  5000  prior  to  the  2003 
revision;  stating  that  the  policies  contained  were  “overly  prescriptive”  and  they  did  not 
“constitute  an  acquisition  policy  environment  fostering  efficiency,  creativity,  and 
innovation”  (26:14).  Hawthorne  then  states  that  the  objectives  of  the  2003  version  are  to, 
“encourage  innovation  and  flexibility;  permit  greater  judgment  in  the  employment  of 
acquisition  principles;  focus  on  outcomes  instead  of  process;  empower  program 
manager’s  to  use  the  system  vice  being  hampered  by  regulation”  (26: 14).  A  couple  of 
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examples  of  those  objeetives  being  eodified  in  the  2003  version  of  the  DoD  5000  series 
are  listed  in  a  Defense  Aequisition  University  (DAU)  briefing  on  DoD  Business 
Transformation.  The  briefing  lists  two  speeifie  ways  that  the  2003  version  will  inerease 
flexibility  in  the  DoD  aequisition  proeess;  one  is  by  allowing  the  program  manager  to 
determine  what  information  is  required  to  satisfy  regulatory  requirements;  and  by 
allowing  the  milestone  deeision  authority  to  tailor  regulatory  requirements  (11:19). 

Over  the  eourse  of  the  DoD  5000  series’  more  than  30-year  history,  it  appears  that 
with  eaeh  iteration,  the  administration  in  power  tried  to  do  what  Seeretary  Wolfowitz 
eited  as  a  reason  for  eaneeling  the  2000  and  2002  version;  whieh  was  to  implement 
proeedures  in  the  instruetions  that  would  foster  an  aequisition  environment  of  “effieieney, 
flexibility,  ereativity,  and  innovation.”  Eaeh  iteration  was  intended  to  improve  the 
proeess  so  the  DoD  eould  proeure  teehnologioally  superior  weapons,  faster  than  any  of 
our  enemies.  Today’s  “box”  environment  is  a  produet  of  eaeh  of  these  iterations  of  the 
DoD  5000  series  that  were  the  result  of  the  politieal  and  world  environments  for  eaeh  of 
their  respeetive  time  periods. 

2.4  Previous  Research 

This  foeuses  on  examining  the  researeh  that  has  been  eondueted  on  the  eost  of  the 
oversight  of  MDAPs.  This  researeh  has  been  eondueted  to  fill  the  gap  in  eurrent 
literature  that  is  pointed  out  in  the  following  statement:  “..  .definitive  evaluative  studies 
do  not  yet  exist  on  the  effieieney  of  these  various  [aequisition]  reforms. . .”  (2:295).  That 
quotation  eame  from  an  artiele  written  by  Maj  Joseph  Besselman,  Ashish  Arora  and 
Patriek  Larkey  whieh  dealt  with  evaluating  the  eost  of  purehasing  styles  in  the  defense 
aequisition  system.  It  was  direeted  at  various  aequisition  reforms  eoming  out  of  the  mid 
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to  late  1990s  including  integrated  product  development  and  the  employment  of 
commercial  practices.  No  previous  literature  specifically  addresses  the  cost  of  the 
vertical  levels  of  oversight  of  MDAPs  that  accrues  while  an  MDAP  moves  from  one 
stage  of  development  to  the  next.  A  look  at  studies  dealing  more  generally  with  the  topic 
of  oversight  of  DoD  acquisition  follows. 

2,4,1  Contractor  Oversight 

In  a  General  Accounting  Office  report  printed  in  1997  titled  Acquisition  Reform: 
DoD  Faces  Challenges  in  Reducing  Oversight  Costs,  the  GAO  reports  on  the  results  of 
“reinvention  laboratories  which  were  conducted  in  ten  different  defense  contractor  sites 
in  1994  with  an  eye  on  reducing  oversight  costs”  (24:1).  This  effort  was  one  of  the  major 
reforms  coming  out  of  the  National  Performance  Review  of  1993.  Each  of  the  test  sites 
set  up  functional  evaluation  teams  consisting  of  members  from  various  different 
government  departments,  including  representatives  from  weapon  systems  program 
offices.  Their  objectives  were  to  perform  cost  benefit  analyses  of  oversight  requirements 
and  eliminate  non-value  added  requirements.  It  was  a  large  undertaking  with  mixed 
results.  The  labs’  work  resulted  in  “limited  progress  in  implementing  changes  to  reduce 
contractors’  costs  of  complying  with  government  regulations  and  oversight  requirements” 
(24:4).  They  concluded  that  although  they  still  firmly  believed  the  initiatives  were 
worthwhile,  great  progress  could  not  be  made  without  greater  support  from  across  the 
DoD.  The  GAO  report  highlights  an  important  part  of  the  cost  of  oversight  of  acquisition 
programs,  however,  it  deals  with  the  cost  of  contractor  oversight,  not  oversight  as  sought 
to  be  evaluated  in  this  thesis. 
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2,4.2  Cost  Overruns 


A  1999  article  in  Acquisition  Review  Quarterly  studied  the  results  of  the 
reeommendations  of  the  Packard  Commission  as  seen  in  aequisition  progress  evaluated 
over  eight  years  from  1988  through  1995.  The  report  used  data  from  the  Defense 
Acquisition  Executive  Summary  (DAES)  database  and  found  that  Paekard  Commission 
initiatives,  “did  not  reduee  the  average  eost  overrun  pereent  experienee  on  269  eompleted 
defense  aequisition  contracts”  (7:251).  This  article  examines  the  effect  the  initiatives  had 
on  eost  overruns,  but  did  not  deal  with  the  eost  of  oversight.  Of  note,  the  study 
eoneluded  that  not  only  were  the  Paekard  initiatives  ineffeetive  in  regards  to  reducing 
cost  overruns,  but  that  overall  eost  performanee  on  the  269  eontracts  they  reviewed 
actually  worsened  (7:258). 

With  the  topie  of  eost  overruns  breaehed  through  the  literature  review  of  the 
previous  article,  the  foeus  moves  from  eost  to  oversight.  Sinee  eost  overruns  don’t  focus 
on  cost  as  defined  in  this  researeh  effort,  is  there  current  literature  that  relates  eost 
overruns  to  oversight?  The  answer  to  that  question  is  a  resounding  yes  as  seen  in  the 
literature  review  of  three  articles. 

In  a  1994  Acquisition  Review  Quarterly  artiele  eost  overruns  experieneed  on 
government  eontraets  are  attributed  in  large  part  either  to  failing  to  ensure  eontrol  (or 
oversight)  is  shared  equitably  or  to  plaeing  too  mueh  eontrol  with  the  entity  that  is 
responsible  for  produeing  the  work,  in  other  words  the  eontraetor  (40:31).  The  author 
begins  his  exploration  by  asserting  the  premise,  “If  you  want  a  job  done  to  your 
standards-meaning  time  required  to  perform  the  work,  the  eost,  the  level  of  quality,  and 
the  required  quantity-you  should  be  in  eontrol,  from  start  to  finish”  (40:30).  He  then 
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makes  recommendations  for  how  the  government  can  gain  more  effective  control  or 
oversight  of  its  defense  acquisition  contracts.  He  first  states  that  the  government  must 
set-up  program  controls  and  then  enforce  them  through  regulations,  reporting 
requirements,  by  giving  necessary  authority  to  responsible  individuals  and  by  breaking 
work  into  manageable  parts  (40:32).  The  author  then  states  that,  “control  by  government 
of  its  programs  can  be  gained  only  by  placing  reporting  and  control  methods  and 
procedures  in  its  contracts”  (40:33).  On  the  topic  of  contracts,  the  author’s  final 
recommendation  for  how  the  government  can  achieve  more  effective  oversight  through 
contracts  is  by  employing  smaller,  task  specific  contracts  to  a  variety  of  competing 
bidders  (40:34).  The  author  presents  the  idea  that  the  governments  has  put  itself  in  the 
position  to  experience  high  cost  overruns  by  employing  large  contracts  to  one  source 
because  by  doing  so,  competition  is  decreased  and  costs  are  increased.  The  author  states 
that  by  employing  competitive,  task  specific  contracts,  competition  would  be  increased 
which  would  in  turn  lower  overall  program  cost  (40:34). 

In  another  1994  article  that  appeared  in  Acquisition  Review  Quarterly  the  topic  of 
cost  overruns  and  oversight,  specifically  program  advocacy  of  persons  in  oversight 
positions  is  covered.  Christensen  presents  the  idea  that  too  much  emphasis  on  program 
advocacy,  or  attempting  to  present  a  program  in  an  over-optimistic  light  despite  contrary 
data,  can  be  dangerous  and  refers  to  the  Navy’s  failed  A-12  acquisition  program  as  an 
example  (estimates  of  the  A-12’s  completion  costs  were  in  some  cases  one  billion  dollars 
higher  than  the  estimates  supported  by  the  government  and  its  contractors)  (6:26).  The 
author  examines  one  form  of  program  advocacy-failing  to  report  accurate  cost  overrun 
data  to  the  right  people  with  the  goal  of  determining  how  widespread  this  form  of 
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program  advocacy  is  in  defense  acquisition  programs.  To  meet  his  objective,  the  author 
examines  64  eompleted  aequisition  eontraets  that  range  that  oeeurred  from  1971-1991 
(6:29).  The  methodology  the  author  used  to  determine  if  advoeaey  was  present  was  to 
eompare  eurrent  eost  overrun  data  at  various  stages  of  eompletion  to  final  program  eost 
overrun  estimates  provided  by  both  the  eontraetor  and  the  government.  If  the  eost 
overrun  estimate,  at  any  stage  of  eompletion  was  less  than  the  final  eost  overrun  estimate, 
than  advoeaey  was  present.  The  author  found  that  on  average,  for  all  64  eontraets,  final 
estimates  of  eost  overruns  for  both  the  eontraetor  and  the  government  were  less  than 
actual  overruns  at  every  stage  of  eompletion,  beginning  at  the  ten  pereent  eompletion 
phase  (6:31).  Of  note,  the  author  also  found  that  the  results  were  not  sensitive  to  eontraet 
type,  eontraet  phase,  weapon  type,  or  whieh  serviee  funetioned  as  the  lead  and  that  on 
average  the  eontraetor  was  more  optimistie  than  the  government  (6:32). 

The  final  example  of  literature  on  eost  overruns  that  relates  to  oversight  eame 
from  a  1998  artiele  in  Acquisition  Review  Quarterly  in  whieh  the  author  sought  to 
identify  key  faetors  for  sueeessful  defense  aequisition  programs.  The  author  uses  eost 
overruns  as  one  of  the  delineators  for  a  sueeessful  program  because  as  he  asserts, 
“Department  of  Defense  aequisition  programs  and  projeets  frequently  experienee  eost 
overruns,  performanee  defieiencies,  sehedule  delays,  or  eaneellation”  (12:35).  The 
methodology  the  author  used  to  determine  suceess  faetors  was  to  first  survey  32  program 
managers  in  whieh  they  would  identify  faetors  they  thought  most  eontributed  to  a 
program’s  sueeess.  The  author  reeeived  18  surveys  baek  and  the  number  one  and  two 
sueeess  factors  (among  several  others)  identified  were  meets  teehnieal  performance 
objeetives  and  works  well  when  fielded  respeetively  (12:37).  Next  the  author  eondueted 
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a  literature  review  to  determine  how  many  times  the  suecess  factors  identified  in  the 
surveys  appeared  in  relevant  literature.  The  success  factors  that  appeared  most  often 
would  be  considered  significant.  Comparing  the  literature  review  to  the  surveys,  the 
author  found  well-defined  requirements  and  quality  people  were  the  most  significant 
success  factors  at  47%  (12:41,42).  The  methodology  presented  in  this  article  used  cost 
overruns  as  a  metric  to  determine  program  success.  The  relationship  between  cost 
overruns  and  oversight  turned  up  in  the  surveys  completed  by  the  eighteen  program 
managers.  On  the  topic  of  oversight  of  their  programs,  program  managers  “. .  .viewed 
involvement  from  support  agencies  and  higher  commands  as  a  hindrance,”  and  surveys 
went  on  to  reveal  that  all  of  the  program  managers  “. .  .felt  that  involvement  of  Congress 
and  GAO  in  specific  programs  was  a  detriment  to  program  success”  (12:38). 

2,4,3  Congressional  Oversight 

A  1995  article 'm  Acquisition  Review  Quarterly  summarized  the  actions  of  a 

project  team  going  through  a  program  management  course.  Their  aim  was  to  review 

congressional  oversight  of  DoD  acquisition  programs  (37:82).  They  specifically  focused 

on  the  reporting  process  in  order  to  find  areas  for  improvement.  The  team  only  had  six 

weeks  to  complete  the  project  and  intensively  interviewed  members  from  both  House  and 

Senate  Congressional  staffs,  DoD  Comptroller,  and  other  pertinent  government  agencies. 

The  results  of  their  study  were  interesting.  Despite  paperwork  streamlining  efforts 

through  reform  and  as  implemented  in  various  versions  of  the  DoD  5000  series, 

DoD  reports  to  Congress  grew  224%  from  1980  to  1988,  far  faster  than  any 
other  government  agency  and  nearly  three  times  the  average  growth  of  other 
agencies.  Acquisition  issues  comprise  approximately  45%  of  the  reports 
requested  by  Congress.  (37:84-85) 
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This  study  produced  definitive  results  and  dealt  with  the  topie  of  oversight; 
however,  it  dealt  with  eongressional  oversight  and  did  not  deal  speeifieally  with  oversight 
eosts.  One  might  be  able  to  assoeiate  oversight  eost  growth  with  the  inerease  in  the 
number  of  reports  required.  However,  the  artiele  did  not  explore  that  premise. 

2.4.4  Some  Evidence  of  Progress 

Another  look  at  literature  relating  to  oversight  of  MDAPs  was  found  in  a  1996 
issue  of  Program  Manager  Journal.  One  artiele  featured  a  speech  by  then 
Underseeretary  of  Defense  for  Aequisition  and  Teehnology,  Dr.  Paul  G.  Kaminski  given 
to  kiek  off  Aequisition  Reform  Aeeeleration  Day,  31  May  1996  (28:28).  On  this  day,  the 
Defense  Aequisition  Community  offieially  stood  down  to  evaluate  current  progress  on 
implementing  reform  initiatives.  One  item  Under  Seeretary  Kaminski  addressed  dealt 
with  the  number  of  meetings  required  to  reaeh  a  milestone  event.  Dr.  Kaminski  stated, 
“beeause  our  early  and  eontinuous  insight  proeess  is  helping  resolve  major  issues,  I  have 
been  able  to  eancel  numerous  formal  DAB  meetings,”  he  went  on,  “last  year,  26  DAB 
meetings  were  seheduled  to  oeeur  but  I  only  had  to  eonvene  eight  of  them”  (28:30).  This 
speeeh  in  itself  did  not  eompletely  address  the  topie  of  oversight  costs;  however  it  did 
reveal  evidenee  that  aequisition  reform  has  resulted  in  reduetion  of  the  number  of 
meetings  neeessary  for  a  milestone  event. 

2.4.5  Oversight  and  Review  Process  Action  Team  (ORPAT) 

The  Program  Manager  Journal  featured  an  artiele  in  its  May- June  1995  edition  in 
whieh  U.S.  Army  Colonel  John  S.  Caldwell,  Jr.  was  the  subjeet  of  an  interview.  His 
eomments  were  important  beeause  of  his  role  as  team  lead  of  the  Defense  Aequisition 
Reform  Oversight  and  Review  Proeess  Aetion  team  that  was  eondueted  from  7 
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September  through  16  December  1994  (3:2).  The  team’s  mission  was  .  .to  develop  with 
in  90  days  a  comprehensive  plan  to  reengineer  the  oversight  and  review  process  for 
systems  acquisition. . .”  (3:2).  Some  of  the  recommendations  of  the  team  were 
groundbreaking,  including:  creating  a  three  milestone  process  (the  current  process 
employs  three  milestones);  decreasing  the  number  of  documents  required  for  each 
milestone  decision;  dramatically  reducing  the  number  of  formal  pre  milestone  meetings 
occurring  in  order  to  prepare  for  a  milestone  decision  event;  and  making  IPX’s  part  of  the 
formal  process  to  conduct  oversight  (3:5-6).  Many  of  the  recommendations  developed 
from  the  work  the  team  conducted  in  1994  are  seen  in  today’s  acquisition  process.  This 
work  did  look  at  oversight  as  defined  for  the  purpose  of  this  research  effort,  but  unlike 
the  other  studies,  this  study  did  evaluate  the  cost  of  that  oversight. 

Colonel  Caldwell’s  ORPAT  team  had  90  days  to  put  recommendations  for 
streamlining  defense  acquisition  together  in  a  report  to  the  DoD.  In  addition  to  the 
recommendations  from  Col  Caldwell’s  team,  they  also  established  an  estimate  for  the 
cost  of  oversight  of  acquisition  programs.  Col  Caldwell’s  team  took  a  look  at  three  joint 
service  acquisition  programs  an  attempted  to  come  up  with  an  estimated  cost  of  oversight 
and  review.  Their  analysis  resulted  in  an  average  estimate  of  $10-12  million  for  a  single 
milestone  and  an  estimate  of  $40-50  million  for  an  entire  joint  acquisition  program 
(14:9).  The  report  recognizes  the  fact  that  $50  million  is  a  small  percentage  of  a  billion 
dollar  program  however,  since  the  failure  of  one  defense  program  could  mean  the  loss  of 
American  lives  in  the  field,  oversight  counts  for  a  lot  more  than  the  percentage  shows. 
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2.5  Summary 

This  chapter  presented  the  guidelines  under  whieh  a  “box”  program  must  operate, 
went  on  to  discuss  the  evolution  of  the  regulatory  series  that  establishes  those  guidelines, 
and  concluded  with  a  look  at  current  studies  dealing  with  the  topie  of  oversight  of 
defense  aequisition  programs.  It  is  apparent  from  the  findings  (or  the  lack  thereof)  that 
there  are  very  few  definitive  studies  in  whieh  oversight  costs  (as  defined  in  this  ehapter) 
are  evaluated.  In  the  ORPAT’s  report  to  the  DoD  however,  we  found  some  hope  with 
their  estimated  oversight  and  review  eost.  The  goal  of  the  eurrent  research  is  to  extend 
prior  research  by  evaluating  the  cost  of  oversight  of  the  DoD’s  MDAPs 
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3,0  Methodology 


3.1  Overview 

From  the  previous  two  ehapters,  we  now  have  a  clear  picture  of  the  focus  of  this  research. 
This  research  will  ultimately  estimate  the  cost  of  oversight  of  Major  Defense  Acquisition 
Programs  for  those  programs  strictly  under  the  Direction  of  the  DoD  5000  series  of 
instructions.  In  addition  to  estimating  the  cost  of  oversight,  the  aim  of  this  research  will 
be  to  answer  specific  research  questions.  The  first  and  most  important  question  to  be 
answered  is  what  is  the  cost  of  oversight  for  “box”  programs?  The  next  question  is  how 
does  the  cost  of  oversight  for  box  programs  compare  to  the  cost  of  oversight  for  MDAP’s 
operating  under  a  different  framework;  specifically,  communication  acquisition  programs 
which  are  operating  in  a  “virtual  box”  and  space  acquisition  programs  which  are 
operating  outside  the  box?  The  final  research  question  to  be  answered  by  this  research  is 
what  are  the  key  drivers  that  affect  the  cost  of  oversight  of  MDAP’s?  As  noted  in  chapter 
2,  the  Delphi  Method  of  surveying  experts  will  be  employed  to  answer  these  questions. 
This  chapter  will  outline  what  the  Delphi  Method  is,  how  it  works,  and  how  specifically 
the  Delphi  Method  will  be  utilized  for  this  research  in  order  to  answer  the  three 
aforementioned  research  questions. 

3.2  Delphi  Method  Background 

In  this  section,  of  the  chapter,  some  background  information  on  the  Delphi 
Method  is  provided.  After  discussing  the  history  of  the  Delphi  Method,  it  is  important  to 
discuss  what  it  is,  and  finally  describes  how  it  works.  After  discussing  the  history  and 
providing  a  thorough  background,  the  Delphi  Method  will  prove  itself  as  a  perfect  fit  and 
the  chapter  will  close  with  why  the  Delphi  Method  was  the  chosen  methodology  for  this 
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research.  The  methodology  for  the  execution  of  this  current  research  project  will  be 
interspersed  within  the  description  of  each  of  these  subject  areas. 

3.2.1  History  of  the  Delphi  Method 

According  to  Clayton,  the  name  “Delphi”  was  associated  with  Greek  mythology 
and  refers  to  a  Delphi  Oracle  which  was  capable  of  predicting  the  future  (8:376).  The 
Delphi  Method  was  actually  born  in  the  1960s  out  of  the  American  defense  industry  as 
part  of  a  project  called  “Project  Delphi”  which  was  a  study  conducted  by  the  RAND 
Corporation  in  support  of  an  exploration  by  the  U.S.  Air  Force  (4:700-701).  The  U.S.  Air 
Force  wanted  to  determine  what  would  be  key  nuclear  targets  and  what  would  be  the 
likely  number  of  warheads  employed  against  the  United  States  in  the  event  of  nuclear 
attack  by  the  Soviets.  “Project  Delphi”  sought  to  reach  a  consensus  of  expert  opinion  in 
order  to  answer  those  two  critical  questions  from  the  viewpoint  of  a  Soviet  nuclear 
strategist. 

3.2.2  What  is  the  Delphi  Method? 

The  Delphi  Method  is  best  described  as  a  communication  tool  that  facilitates  a 
communication  process  by  allowing  a  group  of  individuals  to  work  as  a  whole  to  deal 
with  a  problem  (4:701).  The  Delphi  Method  attempts  to  reach  a  consensus  of  opinion 
among  the  members  of  the  group,  which  will  here  on  be  referred  to  as  an  expert  panel, 
through  a  series  of  questionnaires.  A  key  element  of  the  questionnaires  is  that  they  are 
completed  anonymously  to  allow  for  freedom  of  expression  and  then  collected, 
summarized  and  returned  to  panel  members  to  give  them  the  opportunity  to  refine 
original  responses  with  the  added  benefit  of  knowing  the  rest  of  the  panel  members’ 
responses.  This  process  is  continued,  “until  consensus  is  obtained  or  the  law  of 
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diminishing  returns  sets  in”  (26:1010).  Another  key  element  of  the  proeess  is  that  the 
panel  is  made  up  of  pre-selected  experts  who  never  physically  have  to  be  in  the  same 
location.  The  process,  which  came  into  practice  in  the  1960s,  could  only  be  conducted  by 
traditional  mail,  but  of  course  can  now  be  conducted  via  the  web  or  e-mail,  or  a 
combination  of  both. 

3.2,3  How  the  Delphi  Method  Works 

The  previous  section  of  this  chapter  offered  a  preliminary  look  at  how  the  Delphi 
Method  works,  but  this  section  will  go  into  much  greater  detail  on  the  workings  of  the 
Delphi  Method.  First,  it  is  important  to  answer  some  questions.  The  first  is  why  use  a 
panel  of  experts  that  never  meet  instead  of  just  a  single  expert.  The  reason  is  that  an 
individual  is  operating  along  which  means  they  could  forget  something  or  fail  to  consider 
an  issue.  Clayton  highlighted  this  issue  when  he  discussed  the  fact  that  individuals  don’t 
get  the  benefit  of  hearing  the  ideas  of  others  so  that  they  can  perhaps  refine  their  ideas 
(8:375).  Clayton  goes  on  to  state  that  by  combining  the  judgment  of  a  large  number  of 
people,  there’s  a  better  chance  of  arriving  at  the  truth. 

Having  explained  why  a  separated  group  and  not  an  individual,  the  question  then 
becomes,  if  a  group  is  better  than  an  individual,  wouldn’t  it  be  better  to  put  them  in  a 
room  together  to  allow  them  to  brainstorm  and  hammer  out  a  consensus?  Though  this 
research  operates  under  Clayton’s  premise  that  the  shared  ideas  of  a  group  of  experts  is 
better  than  a  single  expert,  putting  a  panel  in  a  room  together  could  lead  to  group  think 
(8:375).  This  phenomenon  is  the  result  of  a  few  dominant  personalities  controlling  the 
discussion  and  potentially  strong  arming  a  consensus  despite  the  initial  objections  of 
possibly  better  informed,  yet  more  timid  panel  members. 
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Now  that  the  two  preliminary  questions  regarding  the  overall  set  up  of  the  Delphi 
Method  have  been  answered,  the  next  step  is  to  deseribe  the  workings  of  the  Delphi 
Method.  To  aid  in  this  presentation,  the  key  elements  of  the  workings  of  the  Delphi 
Method  are  explained  best  in  Figure  3.1. 


Figure  3.1:  Steps  in  the  Delphi  Proeess  (13:2) 

First,  a  problem  is  defined.  For  this  research,  the  research  questions  are  the  main 
problems  defined  which  is  to  determine  the  cost  of  oversight  for  “box”  programs  as  well 
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as  compare  the  eosts  of  those  “box”  programs  to  the  virtual  and  space  programs.  The 
other  research  focus,  using  the  Delphi  Method  is  to  determine  key  oversight  cost  drivers. 
The  next  step  is  to  develop  a  questionnaire  that  is  speeifie  enough  to  divulge  the  data 
necessary  to  answer  those  questions.  The  following  step  is  to  select  a  panel  of  experts  to 
answer  the  questionnaire.  The  questionnaires  are  then  sent  to  the  panel  and  when 
eompleted  they  are  colleeted,  analyzed,  and  summarized.  If  consensus  is  not  reached,  the 
summarized  responses  will  then  be  sent  baek  out  to  the  panel  to  allow  them  to  rethink  the 
questions  now  that  they  have  the  added  benefit  of  the  input  from  the  other  group 
members.  This  proeess  of  sending  out  the  questionnaires  and  then  getting  them  back  and 
analyzing  them  continues  in  a  looping  pattern  and  each  loop  is  referred  to  as  a  “round.” 
Eaeh  time  a  new  questionnaire  is  distributed  marks  the  beginning  of  a  new  round.  The 
number  of  rounds  is  determined  by  the  achievement  of  eonsensus  of  the  expert’s 
opinions.  Early  criticisms  of  the  Delphi  Method  centered  on  the  fact  that  originally,  (due 
to  lack  of  technology)  questionnaires  were  sent  by  traditional  mail  channels  and 
depending  on  the  number  of  rounds  needed  to  aehieve  eonsensus,  the  proeess  took  from 
several  months  up  to  a  year  or  two  to  complete.  Today’s  technology  enables  the  proeess 
to  flow  mueh  more  quickly,  and  for  the  purpose  of  this  partieular  research  effort,  all 
eommunication  during  the  proeess  will  be  conducted  via  e-mail.  Chou  takes  this  e-mail 
centered  Delphi  methodology  a  step  further  by  conducting  a  web  based  Delphi  Process 
whereby  panel  members  and  the  survey  director  interact  in  a  shared  web  program  (5:233- 
236).  In  summary,  the  Delphi  Method,  as  employed  in  this  research  effort,  will  aet  as  a 
eommunication  facilitator  that  attempts  to  achieve  a  consensus  of  opinions  from  an 
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anonymous,  geographically  separated  panel  of  experts  through  a  series  of  questionnaires 
all  conducted  via  e-mail. 

3,2,3, 1.  The  Rounds  of  the  Delphi  Method 

As  previously  mentioned,  each  time  a  questionnaire  is  distributed  to  panel 
members  and  returned  to  the  person  directing  the  research  effort  constitutes  a  round  of 
the  Delphi  Method.  The  big  question  that  arises  deals  with  how  many  rounds  of  the 
Delphi  are  necessary  to  ensure  the  data  is  stable.  Clayton  states  that  only  four  phases  are 
needed  and  that  the  final  round  is  sent  out  to  “provide  reasons  as  to  why  they  agree  or 
disagree  with  the  final  results”  (8:129).  Chan  et.  al  agreed  in  their  study  by  establishing 
four  rounds  (4:701)  However,  Ludwig  states  that  “Delphi  rounds  continue  until  a 
predetermined  level  of  consensus  is  reached  or  no  new  information  is  gained”  (31:3). 
While  a  study  in  Scotland  by  Dr.  Kerr  limited  the  number  of  rounds  to  3.  (29:3)  In 
recent  nursing  research,  Hasson  et.  al  limited  the  number  of  rounds  depending  on  “time 
available. . .”  (25: 101 1).  The  research  did  not  find  a  specific  number  of  rounds  needed. 
Most  researchers  using  the  Delphi  Method  set  the  criteria  of  consensus  and  time  available 
while  some  limited  on  a  firm  number.  Based  on  the  evidence,  the  Delphi  method  as 
employed  in  this  research  effort  to  answer  the  research  questions,  will  contain  a  minimum 
of  two  rounds  and  a  maximum  of  four. 

3,2,3,2  Delphi  Method  Questionnaires 

Mitchell  goes  into  great  detail  outlining  the  construction  and  administration  of  the 
Delphi  questionnaires.  He  clearly  outlined  the  length  the  questionnaire  should  be  by 
stating  how  long  it  should  take  each  panel  member  to  complete  the  questionnaire.  On 
this  topic  he  states  that  the  questionnaire  should  take  no  longer  than  30  minutes  to 
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complete  (32:345).  The  basis  for  this  assertion  is  his  own  experienee  as  he  goes  on  to 
state  that  there  have  been  no  empirieai  studies  eondueted  on  the  appropriate  length  of 
time  to  eomplete  a  Delphi  questionnaire.  Mitehell  also  diseusses  the  eonstruetion  of  the 
questionnaire  for  eaeh  round  of  the  Delphi  Method.  He  states  that  questions  should  be 
elearly  stated  and  should  not  be  identieal  from  round  to  round  beeause  the  repetition 
eould  eause  partieipant  boredom,  whieh  eould  hamper  results  (32:342).  Clayton  also 
diseussed  the  format  of  the  questionnaires  on  a  round  by  round  basis.  He  states  that 
round  one  questionnaires  should  be  elearly  worded  but  allow  for  the  most  freedom  in 
responses.  Round  one  responses,  onee  eolleeted,  should  be  turned  into  generie 
statements  summarized  with  measures  of  eentral  tendeney  and  then  resent  to  panel 
members  to  begin  Round  two.  In  round  two,  the  proeess  of  seeking  eonsensus  begins. 

To  aid  in  the  quest  for  eonsensus  panel  members  that  wish  to  ehange  previous  responses 
must  provide  reasons  for  doing  so.  In  round  three  and  subsequent  rounds,  questionnaires 
should  summarized  responses  with  a  summary  of  reasons  for  ehanging  responses  and  this 
proeess  eontinues  until  eonsensus  is  met  (8:378).  The  questionnaires  in  support  of  this 
researeh  effort  will  be  eonstrueted  aeeording  to  the  proeedures  outlined  by  Clayton  and 
Mitehell.  The  number  of  questions  will  be  limited  to  ten  or  less.  The  maximum  amount 
of  time  needed  to  eomplete  eaeh  questionnaire  is  estimated  at  20  minutes.  Eaeh  returning 
questionnaire’s  questions  are  altered  in  eaeh  round  based  on  the  previous  round’s  input. 
This  will  ensure  eaeh  panel  member  has  the  opportunity  to  re-evaluate  eaeh  question. 
3,2,3,3  Delphi  Method  Consensus 

The  rounds  of  questionnaires  must  eventually  eome  to  a  elose.  In  order  to  set  the 
parameters  prior  to  beginning,  onee  eonsensus  is  reaehed,  the  rounds  will  diseontinue. 
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Webster’s  New  International  Unabridged  Dictionary  defines  consensus  as,  “unanimity  or 
general  agreement  in  matters  of  opinion”  (42:567).  If  that  definition  is  applied  to  the 
Delphi  Method  as  employed  in  this  research  effort,  once  the  panel  reaches  a  majority 
opinion,  the  process  is  complete,  but  just  a  majority  may  not  be  far  enough.  Simply 
operating  under  the  theme  of  “majority  rule”  could  overlook  important,  though  less 
frequently  occurring  opinions.  Therefore,  in  terms  of  the  application  of  the  Delphi 
Method  for  this  research  effort,  consensus  must  be  defined.  The  problem,  as  Williams 
and  Webb  state,  “Consensus  is  poorly  explained  in  studies  which  use  the  Delphi 
technique...”  (43:182).  Hasson  et.  al.  also  state  that  “A  universally  agreed  proportion 
does  not  exist  for  the  Delphi. . .”  (25: 101 1).  Hasson  et  al.  does  list  various  studies  who 
established  percentages  for  defining  consensus,  but  all  vary  dramatically  and  result  in 
mostly  a  straight  majority  rules.  This  study  completed  by  Schiebe  et.  al.  recommends 
stability  of  responses  throughout  the  rounds  as  a  better  indicator  of  consensus  by 
evaluating  the  changes  in  the  questions  to  a  quartile  in  a  distribution  (36:IV:C).  Without 
much  empirical  evidence  to  support  a  concrete  definition  of  consensus,  this  research 
effort  will  take  an  approach  similar  to  the  one  recommended  by  Schiebe  et.  al.  Each 
question  will  be  evaluated  on  the  response  and  as  answers  become  stable,  the  question 
will  be  considered  “closed”  until  all  questions  are  closed  or  four  rounds  have  been 
completed. 

3.2,3,4  Delphi  Method  Expert  Panel 

Another  obstacle  when  performing  the  Delphi  Method  is  deciding  how  big  the 
expert  panel  should  be.  Spinelli  conducted  research  utilizing  the  Delphi  Method  and  the 
panel  consisted  of  “24  key  influential  persons  knowledgeable  as  to  the  factors  influencing 
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the  general  environment. . .)  (38:74).  Ludwig  eonducted  research  but  had  a  different 
approach  to  establishing  a  panel.  Ludwig  stated  that  “The  number  of  respondents  was 
generally  determined  by  the  number  required  to  constitute  a  representative  pooling  of 
judgements  and  the  information  summarizing  capability  of  the  research  team”  (3 1 :2). 

This  establishes  the  precedent  that  as  long  as  all  members  of  the  focus  research  is 
represented,  the  number  of  members  on  the  panel  is  up  to  the  researcher.  Ludwig  then 
states  “The  majority  of  Delphi  studies  have  used  between  15-20  respondents  and  run  over 
periods  of  several  weeks”  (31:2).  Since  it  seems  difficult  to  find  15-20  volunteers  for  this 
research,  further  studies  were  scanned  and  established  more  attainable  precedents.  Chan 
et.  al.  stated  in  their  selection  process  “The  ten  members  of  the  panel  represent  a  wide 
distribution  of  professional  people. . .”  (4:701).  Another  study  by  Des  Marchais  reduced 
the  panel  size  to  six  (17:504).  Overall,  William  and  Webb  summarize  the  panel  selection 
methodology  by  stating  “First,  there  is  no  agreement  regarding  the  size  of  the  panel,  nor 
any  recommendations  concerning  sampling  techniques”  (43:182) 

The  panel  assembled  to  answer  the  research  questions  posed  in  this  thesis  will  be 
of  the  heterogeneous  type  and  will  embody  the  principal  of  breadth  of  members’ 
experience  while  maintaining  the  similar  target  career  field.  The  panel  will  contain  a 
minimum  of  five  and  a  maximum  of  ten  members. 

Once  the  size  of  the  panel  has  been  decided,  establishing  criteria  to  judge  who  are 
the  experts  is  needed.  Based  on  the  findings  that  were  a  result  of  the  research  conducted 
to  complete  this  chapter,  it  appears  there  is  no  clear  cut  definition  of  what  constitutes  an 
expert.  While  discussing  the  topic  of  expert  panel  member  selection,  Mitchell  states,  “No 
reported  Delphi  study  has  addressed  this  selection  issue”  (32:340).  Dawson  and  Brucker, 
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in  their  research,  summarized  the  criteria  for  determining  experts  used  in  several  Delphi 
studies  in  their  field.  The  common  theme  was:  general  experience  of  seven  years; 
specific  experience  of  five  years;  at  least  one  published  article;  at  least  one  national 
conference  presentation;  and  experience  should  be  recent  to  within  the  last  three  years 
(10:132-134).  For  the  purpose  of  this  research,  we’ll  relax  those  general  standards  a  bit 
by  requiring:  general  experience  of  five  years;  specific  experience  of  two  years;  recent 
experience  within  the  last  five  years;  and  no  qualification  of  presentations  or 
publications. 

Once  the  expert  panel  is  formed,  but  prior  to  the  process  starting,  a  plan  must  be 
instituted  for  panel  attrition.  In  a  study  by  Chan  et  al.  conducted  in  the  field  of  medicine, 
they  achieved  a  response  rate  of  80%  and  went  on  to  state  that  derived  from  various 
studies  that  the  average  response  rate  for  the  medical  field  ranged  from  58%  to  80% 
(4:708).  Mitchell  states  that,  “High  rates  of  attrition  may  mean  that  final  results  are  based 
upon  an  unrepresentative  sub  sample  of  the  original  sample”  (32:341).  To  combat  panel 
attrition  and  the  resulting  degraded  response  rates,  this  research  effort  will  choose  experts 
from  different  but  related  fields  and  have  at  least  one  backup  expert  for  every  expert  so  in 
the  end,  even  with  an  attrition  as  high  as  50%,  all  groups  will  be  represented  and  the  bias 
that  Mitchell  describes  will  be  avoided. 

3.3  Uses  of  the  Delphi  Method 

The  Delphi  Method  has  had  many  uses  in  research.  According  to  the  book  The 
Delphi  Method:  Techniques  and  Applications,  the  Delphi  Method  was  principally  used  as 
a  forecasting  tool  back  as  early  as  the  1960s  and  went  on  to  say  today  the  Delphi  Method 
is  used  for:  normative  forecast;  to  ascertain  values  and  preferences;  quality  of  life 
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estimates;  simulated  and  real  deeision  making;  and  inventive  planning.  The  book  also 
went  on  to  state  that  the  Delphi  Method  is  used  extensively  where  “judgmental  input 
data”  is  needed  when  other  data  is  unavailable  or  too  eostly  (30:615).  Hasson  et  al.  stated 
that  the  Delphi  Method  is  used  frequently  in  health  and  soeial  seienees  (25:1008). 
MitehelTs  artiele  eites  a  table  listing  the  use  of  the  Delphi  method  by  pereentage  by  field 
of  study  from  a  total  of  800  studies.  Delphi  was  most  heavily  used  in  physieal  seienees 
and  engineering  (26%  of  all  studies  eondueted)  and  the  seeond  most  frequent  usage  was 
in  business  and  eeonomies  (23%)  (32:334). 

3.4  Criticisms  of  the  Delphi  Method 

If  employed  properly,  the  Delphi  Method  is  an  exeellent  tool  for  gathering  data  to 
answer  questions  when  that  data  first  appears  to  be  unavailable.  Sinee  this  researeh  effort 
originally  sought  to  analyze  historieal  data  and  beeause  that  data  was  unavailable,  the 
Delphi  Method  appeared  to  be  a  suitable  baekup  method.  There  are  eritieisms  to  bear  in 
mind  before  using  the  Delphi  Method.  The  first  eritieism  deals  with  who  aetually  deeides 
what  qualifies  as  an  “expert”.  Clayton  aeknowledges  that  expertise  is  not  exaetly 
measurable  however,  he  states  that  the  eriteria  is  really  relative  based  on  the  peers  of  the 
experts.  For  this  researeh  effort,  eriteria  for  panelists  will  be  based  eriteria  found  in  the 
seetion  on  the  expert  panel  found  in  this  ehapter.  Using  Clayton’s  premise  that  experts 
are  deemed  as  sueh  by  their  peers,  the  researeh  will  inelude  a  preliminary  survey  of 
potential  experts.  We’ll  supply  them  with  our  panel  eriteria  and  ask  them  whether  they 
agree  with  eaeh  of  he  eriteria  or  not  and  why. 

Williams  and  Webb  introduee  a  seeond  eritieism  of  the  Delphi  Method  whieh  is 
that  the  researeher’s  analysis  and  summary  of  eaeh  rounds’  responses  eould  introduee 
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bias  into  the  process  (43:182).  That  point  is  well  taken  and  to  combat  that  threat, 
responses  will  be  analyzed  using  basic  statistical  methods  (mean,  median,  standard 
deviation)  to  the  fullest  extent  possible.  Additionally,  because  this  research  will  conduct 
the  Delphi  Method  as  part  of  a  group  project,  there  will  be  more  than  one  set  of  eyes 
analyzing  the  responses,  which  should  also  help  to  keep  the  process  honest. 

A  final  criticism  of  the  Delphi  Method  regards  the  question  of  reliability; 
specifically,  what  evidence  is  out  there  that  proves  the  Delphi  Method  is  reliable.  In 
other  words,  have  studies  been  conducted  that  prove  findings  were  consistent  in  different 
Delphi  experiments  using  similarly  composed  panels  answering  the  same  questions. 
Williams  and  Webb  found  that,  “there  is  no  evidence  that  the  Delphi  Method  is  reliable” 
(43:182).  Hasson  et  al.  support  these  findings  stating  that  their  research  discovered, 
“There  is  no  evidence  of  the  reliability  of  the  Delphi  Method”  (25:1012).  Mitchell  stated 
that  other  studies  have  found  a  high  degree  of  replicability,  which  contradict  criticisms 
that  the  Delphi  Method  is  unreliable  or  unproven  (32:351). 
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3.5  Strengths  of  the  Delphi  Method 

The  strengths  of  the  Delphi  Method  outweigh  the  weaknesses  previously 
mentioned.  First,  the  Delphi  Method  enables  a  group  of  experts  in  geographieally 
separated  loeations  to  work  together  without  the  cost  or  other  logistical  problems 
associated  with  bringing  experts  together  at  a  central  location  (10:129).  Anyone  who  has 
tried  to  put  together  a  major  conference  would  greatly  appreciate  this  strength. 

The  second  strength  focuses  on  the  fact  that  the  Delphi  Method  results  in  a 
consensus  of  opinion  without  the  bias  or  group  think  that  might  result  from  a  roundtable 
process  (43:181).  This  “anonymous  factor”  ensures  all  panel  members  are  equally 
involved  and  all  panel  members  feel  free  to  answer  honestly.  By  this,  the  researcher  has 
the  opportunity  to  receive  uncensored  answers. 

Williams  and  Webb’s  research  also  highlights  the  Delphi  concept  of  conducting  a 
series  of  rounds  to  achieve  consensus  (43:181).  The  series  of  rounds  allows  panel 
members  to  review  the  responses  of  their  fellow  panel  members  and  gives  them  the 
chance  to  reconsider  or  even  alter  their  original  responses  with  the  benefit  of  the  added 
input  of  their  fellow  panel  members.  Conducting  only  one  round  would  destroy  the 
intellectual  synergy  created  by  the  sharing  of  ideas  throughout  the  rounds. 

Finally,  a  criticism  of  the  traditional  Delphi  process  that  evolved  into  a  strength 
for  today  was  that  the  traditional  Delphi  process  took  a  long  time  to  complete.  This  long 
time  period  was  due  to  the  fact  that  it  was  used  in  the  1960s  and  1970s  at  a  time  when 
there  was  no  means  other  than  through  postal  channels  to  conduct  Delphi  rounds.  Chien 
Chou’s  article  highlights  the  final  strength  of  the  Delphi  that  evolved — speed.  Chou 
stated  that  traditional  Delphi  processes  averaged  six  to  twelve  months  from  start  to  finish. 
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but  with  e-mail  and  web-based  Delphi  a  three  round  study  can  be  conducted  in  four 
weeks  (5:236). 

3.6  The  Reason  the  Delphi  Method  was  Chosen 

The  originally  theorized  methodology  for  this  research  effort  was  to  examine  the 
paper  trail  left  by  an  actual  MDAP  going  through  a  milestone  decision  point  i.e.  Meeting 
minutes,  meeting  notes,  sign  in  rosters  to  arrive  at  an  estimated  cost  of  oversight.  Using 
these  documents,  the  ranks  and  number  of  people  at  the  meetings  could  be  ascertained  as 
well  as  the  number  and  duration  of  the  meetings.  This  data  could  then  be  used  to 
estimate  a  cost  of  meetings  based  on  length  of  meeting  and  the  hourly  wages  of  each 
attendee.  The  estimate  for  meeting  costs  at  every  level  of  vertical  oversight  could  then 
be  tallied  to  arrive  at  a  total  estimate  of  the  cost  of  oversight  for  an  MDAP  at  a  certain 
key  decision  point.  The  problem  we  encountered  with  this  methodology  is  lack  of  data. 
We  made  the  mistake  of  assuming  the  meeting  minutes,  notes  and  logs  would  be  readily 
available  when  in  fact  in  some  cases  they  were  nonexistent.  I  needed  to  come  up  with  a 
methodology  that  would  enable  me  to  answer  the  research  questions  without  the 
availability  of  historical  data.  An  article  by  V.W.  Mitchell  which  appeared  in  Technology 
Analysis  &  Strategic  Management,  outlines  why  one  would  use  the  Delphi  Method  with 
the  number  one  reason  listed  being  the  unavailability  of  historical  data  (32:338). 

3.7  Comparative  Analysis  for  Data  Collected 

Once  the  rounds  of  the  Delphi  are  completed,  all  data  from  this  study  must  be 
statistically  compared  with  the  data  collected  by  Neal  for  Command,  Control, 
Communications,  and  Intelligence  (C3I)  and  DeReus  for  Space  Acquisition  Programs. 
After  acquiring  their  data,  all  data  for  questions  two  through  ten  will  be  placed  into  a 
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statistical  analytical  software  package  with  a  graphieal  user  interfaee,  such  as  JMP  5.0.1 
statistical  software.  The  data  will  be  entered,  for  eaeh  question  two  through  ten  in  the 
format  seen  in  Table  3.1. 

Table  3 . 1 ;  Data  Input  for  Statistieal  Analysis 


Oversight 

2-Low 

2-Avg 

2-High 

Space 

1 

4 

6 

Space 

2 

6 

9 

Space 

1 

4 

7 

Space 

2 

4 

8 

DoDD  5000 

6 

8 

16 

DoDD  5000 

8 

12 

18 

DoDD  5000 

6 

10 

18 

DoDD  5000 

6 

10 

25 

DoDD  5000 

6 

12 

18 

C3I 

12 

20 

30 

C3I 

7 

9 

12 

C3I 

12 

20 

30 

C3I 

12 

20 

30 

The  format  in  Table  3.1  will  allow  JMP  5.0.1  to  analyze  the  statistieal  differences  and 
will  provide  a  value  whieh  will  test  whether  or  not  there  is  a  statistieal  differenee  among 
the  different  oversight  proeesses. 

To  eonduet  the  analysis  of  varianees,  eaeh  oversight  proeess  will  be  eompared 
with  one  other  oversight  proeess  at  a  time.  For  example,  DoDD  5000  will  be  compared 
first  with  Spaee  and  then  eompared  with  C3I.  The  null  hypothesis  for  the  test  is  that  there 
is  not  a  statistical  difference  between  the  means  of  the  populations  being  eompared.  The 
alternative  hypothesis  is  that  there  is  a  statistieal  differenee  between  them. 

3.8  Summary 

This  researeh  effort  is  aimed  at  answering  the  following  researeh  questions: 

1 .  What  is  the  eost  of  oversight  for  “Box”  MDAP’s? 

2.  How  does  the  eost  of  oversight  for  “Box”  MDAP’s  eompare  to  the  eost 
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of  oversight  for  “Non  Box”  MDAP’s? 

3.  What  are  the  Cost  Drivers  for  the  Oversight  of  MDAP’s? 

This  ehapter  outlined  exaetly  how  this  researeh  effort  will  answer  those  questions. 
In  summary:  the  researeh  will  eonsist  of  assembling  a  panel  of  five  to  ten  experts  in  the 
field  of  defense  aequisition;  prepare  questionnaires  aimed  at  eolleeting  the  eost  of 
oversight  at  one  key  deeision  point  and  aimed  at  identifying  oversight  eost  drivers;  then 
employ  the  Delphi  Survey  teehnique  of  sending  out  the  questionnaires,  eolleeting, 
analyzing,  summarizing,  and  resending  questionnaires  to  the  panel;  and  eontinue  with  the 
Delphi  rounds  until  a  eonsensus  of  expert  opinion  is  reaehed.  In  Chapter  4,  the  results  of 
eaeh  round’s  questionnaires  will  be  reeorded  and  summarized. 
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4,0  Data  Results 


4,1  Overview 

The  goal  of  chapter  4  is  to  provide  the  results  based  on  the  responses  from  the 
expert  panel  members  for  the  Delphi  Method.  The  first  section  will  present  the 
generalized  demographics  of  the  panel  members  while  still  maintaining  the  members’ 
anonymity.  The  next  section  will  provide  the  results  of  the  survey  for  each  question.  The 
results  will  be  presented  for  one  question  as  it  passed  through  the  four  separate  rounds  of 
the  Delphi  Method.  The  information  provided  will  include  the  initial  answers  for  each 
question  and  how  the  answer  changed  through  the  rounds  of  the  Delphi  Method.  The 
final  section  will  provide  a  review  of  the  change  in  the  standard  deviation  for  each 
question  and  will  conclude  with  the  final  numbers  that  will  be  analyzed  for  the  cost  of 
oversight  and  will  be  used  to  statistically  compare  with  Neal  and  DeReus. 

To  establish  the  cost  of  oversight,  an  algorithm  was  created  which  multiplies  and 
adds  the  respondents’  estimates  together  to  create  low,  average,  and  high  estimates  for 
the  cost  of  oversight.  The  algorithm  works  by  multiplying  pertinent  questions  together. 
To  arrive  at  a  TDY  cost  estimate  questions  two,  three,  and  four  are  multiplied.  Questions 
five,  six,  and  seven  are  multiplied  to  create  a  personnel  cost  estimate.  Questions  eight, 
nine,  and  ten  are  multiplied  together  to  create  a  meeting  cost  estimate.  Finally,  to  arrive 
at  a  total  program  cost  for  one  milestone  decision  point,  the  estimates  for  TDY, 
personnel,  and  meeting  are  added  together.  The  total  program  cost  for  the  low  estimate  is 
then  represented  by  the  following  algorithm: 

3*((Q21ow*Q31ow*Q41ow)+(Q51ow*Q61ow*Q71ow)+(Q81ow*Q91ow*Q101ow)) 
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The  total  program  cost  for  one  milestone  decision  point  is  multiplied  by  “3”  because 
there  are  three  milestone  decision  points.  This  process  is  repeated  for  the  average  and 
high  estimates  as  well. 

4.2  Panel  Selection 

The  goal  of  panel  selection  was  to  gather  experts  in  “inside  the  box”  acquisition 
programs,  but  from  different  viewpoints  in  the  oversight  process.  The  following 
individuals,  listed  on  the  table  below,  were  selected  and  numbered  to  safeguard  their 
anonymity. 
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Table  4.1;  Panel  Selection  Demographics 


Num. 

Militarv/Civilian 

Breadth  of  Experience 

Years  Aca 

Years  Box 

1 

Civilian 

Program  Office;  Cost  Analysis  in 
support  of  DABs/Milestone  reviews; 
Program/Systems  center;  C-17,  C- 
5B, 

F-15,  F-16,  B-2,  F-22 

28 

6 

2 

Civilian 

Program  Cost/Fogistics;  Acquisition 
Policy  and  Procedures;  F-22,  B-2, 

C-5 

16 

8 

3 

Civilian 

Program  Office;  Financial  Mgt;  JSF 

9 

3 

4 

Civilian 

ASC  Fevel  Financial  Mgt  and  Cost 
Analysis;  Tri-Service  Standoff 

Attack  Missile  (TSSAM),  National 
Developmental  Airlift  Aircraft 
(NDAA);  C-130J 

18 

12 

5 

Civilian 

Program  Office,  Financial  Mgt,  Cost 
Analysis;  PEO  level;  B-1,  C-17, 
Advanced  Cruise  Missile 

23 

20 

Since  the  members  are  now  numbered,  the  rest  of  the  results  and  analysis  will 
refer  to  only  the  number  assigned  for  the  Delphi  Method.  As  seen  in  Table  4.1,  there  is  a 
great  deal  of  depth  of  experience  with  an  average  experience  in  defense  acquisition  of 
18.8  years  and  an  average  experience  with  “inside  the  box”  programs  of  9.8  years. 
Breadth  of  experience  is  not  as  strong  with  the  bulk  of  the  experience  at  the  program 
office  or  Aeronautical  Systems  Center  (ASC)  level  though  panelist  number  five  has  a 
number  of  years  at  the  PEO  level.  The  true  “breadth”  of  experience  for  this  panel  comes 
from  the  different  types  of  programs  they’ve  worked  on  and  the  areas  they  worked  in. 

The  panelists’  breadth  of  experience  in  this  area  provides  adequate  heterogeneity  that 
according  to  the  information  provided  in  the  methodology  section,  will  provide  the 
greatest  probability  of  approaching  the  true  answer  of  the  unknown  forecast  we  are  trying 
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to  make  and  compare.  Prior  Internal  Review  Board  permission  was  requested  and 
obtained  for  this  research  and  the  letter  of  approval  can  be  seen  in  the  attachment  section. 


4.3  Question  One 

Stated  from  the  survey,  1.  From  the  Program  Executive  Officer  (PEO)  request 
for  a  Defense  Acquisition  Board  (DAB  to  the  DAB  milestone  approval,  what  are 
the  five  major  cost  drivers  in  the  oversight  process? 

The  goal  of  question  one  was  to  obtain  the  five  key  cost  drivers  that  the 

respondents  felt  drove  the  cost  of  oversight. 

4,3,1  Results  by  Round 

In  round  one,  the  panel  was  asked  to  provide  the  top  five  oversight  cost  drivers  in 
no  particular  order.  With  duplicates  eliminated,  there  were  23  cost  drivers  identified 
in  all.  They  are  listed  below,  in  no  particular  order; 

Table  4.2:  Round  One  Cost  Drivers 

1  Program  is  Mult-Service 

2  Number  of  Technologies  going  into  the  system 

3  Number  of  Systems  the  System  must  interact  with 

4  Milestone  B  (requires  most  documents;  30  to  be  generated  for  review) 

5  Whether  Completely  new  system  or  just  block  upgrade 

6  Collecting  required  data 

7  Updating  program  schedules,  estimates,  test  plans,  etc. 

8  Drafting  charts 

9  Meetings 

10  Rework/redirection 

1 1  Hours  used  to  develop  briefings 

12  TDYs 

13  User  Involvement 

14  Congressional  involvement 

15  Manpower-Civil  Service  and  Military 

1 6  A&AS  Support  Contractors 

17  Office  Furniture  Supplies-if  new  program 

18  Computer  Equipment-if  new  program 

19  Supporting/reconciling  with  the  CAIG 

20  Clinger-Cohen  Act  Compliance 

21  Preparing  SAMP 

22  Command,  Control,  Communication  Computers,  and  Intelligence  Support  Plan  (C4ISP) 

23  Flight  Test  reports 
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For  round  two,  the  23  items  were  sent  out  and  panel  members  were  allowed  to 
piek  five  drivers  by  plaeing  an  asterisk  in  front  of  the  items  they  thought  were  the  biggest 
eost  drivers.  Nine  drivers  fell  off  the  list  based  on  their  failure  to  reeeive  any  votes.  The 
round  two  results  are  listed  below,  again,  in  no  particular  order: 

Table  4.3:  Round  Two  Cost  Drivers 


#  of  Votes 
Received 

1  Program  is  Mult-Service  4  Votes 

2  Number  of  Technologies  going  into  the  system  2  Votes 

3  Number  of  Systems  the  System  must  interact  with  2  Votes 

4  Milestone  B  (requires  most  documents;  30  to  be  generated  for  review)  2  Votes 

5  Whether  Completely  new  system  or  just  block  upgrade  3  Votes 

6  Collecting  required  data  1  Vote 

7  Updating  program  schedules,  estimates,  test  plans,  etc.  1  Vote 

8  Meetings  1  Vote 

9  Congressional  involvement  2  Votes 

10  Supporting/reconciling  with  the  CAIG  2  Votes 

11  Clinger-Cohen  Act  Compliance  1  Vote 

12  Preparing  SAMP  2  Votes 

13  Command,  Control,  Communication  Computers,  and  Intelligence  Support  Plan  (C4ISP)  1  Vote 

14  Flight  Test  reports  1  Vote 


For  round  three,  the  14  items  were  sent  out  with  the  same  instructions  as  those 
from  round  2.  This  time,  two  items  dropped  off  based  on  receiving  no  votes. 

The  results  from  round  three  are  listed  below: 
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Table  4.4;  Round  Three  Cost  Drivers 


#  of  Votes 
Received 

1  Program  is  Mult-Service  5  Votes 

2  Number  of  Technologies  going  into  the  system  3  Votes 

3  Number  of  Systems  the  System  must  interact  with  2  Votes 

4  Milestone  B  (requires  most  documents;  30  to  be  generated  for  review)  2  Votes 

5  Whether  Completely  new  system  or  just  block  upgrade  4  Votes 

6  Collecting  required  data  1  Vote 

7  Updating  program  schedules,  estimates,  test  plans,  etc.  1  Vote 

8  Meetings  1  Vote 

9  Congressional  involvement  2  Votes 

10  Supporting/reconciling  with  the  CAIG  1  Vote 

11  Preparing  SAMP  2  Votes 

12  Command,  Control,  Communication  Computers,  and  Intelligence  Support  Plan  (C4ISP)  1  Vote 


Since  the  goal  of  question  one  was  to  identify  only  the  top  five  drivers,  the  drivers 
from  the  round  three  results  that  reeeived  only  one  vote  were  eliminated  from  the  fourth 
survey  that  went  out.  With  only  one  round  to  go,  eliminating  the  drivers  with  the  fewest 
votes  was  determined  to  be  the  best  way  to  narrow  the  list  down  to  one  that  would  meet 
the  goal  of  identifying  just  five  drivers.  In  order  to  ensure  the  list  of  drivers  was 
prioritized,  panelists  were  asked  to  identify  the  top  five  drivers  from  the  list  by  plaeing  a 
one  through  five  in  front  of  their  five  drivers;  a  one  in  front  of  the  biggest  driver  down  to 
a  five  for  the  smallest. 

The  results  from  round  four  are  listed  below: 

Table  4.5:  Round  Four  Cost  Drivers 


Drivers  Picked  #  Votes 

1 .2.1 .1 .1  Program  is  Mult-Service  5 

2.1. 5.3.2  Whether  Completely  newsystemorjust  block  upgrade  5 

5,3,4,2,4  Number  of  Technologies  going  into  the  system  5 

4.2.5  Number  of  Systems  the  System  must  interact  with  3 

5.3.5  Milestone  B  (requires  most  documents;  30  to  be  generated  for  review)  3 

3,4  Congressional  involvement 

4,3  Preparing  SAMP 
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To  arrive  at  a  final  top  five  list  of  cost  drivers,  the  two  drivers  that  received  only  two 
votes  were  eliminated  from  the  list.  The  only  task  that  remained  was  to  prioritize  the  list 
of  five  cost  drivers.  The  list  was  prioritized  by  creating  a  simple  average  by  adding  the 
values  of  the  votes  and  dividing  by  the  number  of  votes.  The  final  results  of  the 
prioritization  process  are  below: 

Table  4.6:  Prioritized  List  of  Top  Five  Cost  Drivers 


Drivers  Picked 

1 .2. 1 . 1 . 1  Program  is  Mult-Service 

2. 1 .5.3.2  Whether  Completely  new  system  or  just  block  upgrade 
5,3,4,2,4  Number  of  Technologies  going  into  the  system 

4.2.5  Number  of  Systems  the  System  must  interact  with 

5.3.5  Milestone  B  (requires  most  documents;  30  to  be  generated  for  review) 


Avg 

#  Votes  Score  Rank 

5  1.20  1 

5  2.60  2 

5  3.60  3 

3  3.67  4 

3  4.33  5 


Once  the  scores  were  averaged,  the  drivers  were  ranked  and  as  seen  from  the  above  chart 
we  were  able  to  answer  research  question  number  three.  The  panel  determined  that  the 
biggest  oversight  cost  driver  for  an  acquisition  program  is  if  it  is  a  multi-service  program. 

4.4  Question  Two 

From  the  PEO  recommendation,  to  the  DAB  approval  of  the  milestone,  use 
your  professional  judgment  and  estimate  how  many  TDYs  are  taken  by  one 
person  to  get  one  program  through  one  Milestone. 

The  goal  of  question  two  was  to  find  out  how  many  TDYs  are  taken  by  one 

individual  in  one  program  to  get  through  one  milestone.  The  members  were  asked  to 

provide  a  low,  high  and  average,  or  most  likely  occurrence  for  this  portion.  This  will 

allow  us  to  establish  a  triangular  distribution  that  will  be  used  later  for  the  data  analysis 

portion,  as  well  as  to  allow  us  to  estimate  the  low,  average,  and  high  costs  of  oversight 

for  our  comparison  of  the  three  different  MDAP  processes.  Question  two  sets  up  our 

initial  number  in  our  algorithm  to  calculate  the  first  portion  of  our  cost  of  oversight 
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model.  Question  two,  three,  and  four  will  be  multiplied  to  establish  our  travel  estimate 
for  the  eost  of  oversight. 

4,4,1  Question  Two-  Low  Estimate 

For  the  low  estimate,  round  one  resulted  in  a  pretty  wide  range  from  an  estimate 
of  three  TDYs  to  12  TDYs,  but  the  range  quiekly  elosed  in  and  was  set  by  round  three. 
The  median  was  set  early  on  at  an  estimate  of  six  TDYs.  The  Mode  ehanged  in  round 
two,  but  it  is  elear  that  the  panel  gravitated  around  an  estimate  of  6  TDYs  per  person 
involved  to  get  a  program  through  one  milestone.  Consensus  was  not  reaehed,  however 
the  standard  deviation  was  minimized  to  a  value  of  less  than  one  by  round  4.  The  results 
are  listed  in  Table  4.7,  shown  below. 


Table  4.7:  Question  Two-  Low  Estimate  Results  By  Round 


Frequency 


Round 

Range 

(Member  1,2, 3, 4, 5) 

Mean 

Median 

Mode 

Std  Dev 

1 

3  to  12 

7.00 

6.00 

6.00 

3.32 

2 

4  to  8 

4, 8, 6, 6, 8 

6.40 

6.00 

8.00 

1.67 

3 

6  to  8 

6, 8, 6, 6, 8 

6.80 

6.00 

6.00 

1.10 

4 

6  to  8 

6, 8, 6, 6, 6 

6.40 

6.00 

6.00 

0.89 

4,4,2  Question  Two-  Average  Estimate 

The  average,  or  most  likely,  estimate  was  similar  to  the  low  estimate.  The  range 
started  out  broad,  but  narrowed  in  seope  as  the  rounds  eontinued.  The  median  remained 
eonstant  throughout  the  proeess,  but  it  was  interesting  that  the  mode  or  most  frequently 
oeeurring  estimate  inereased  from  10  to  12.  Looking  baek  at  the  eolumn  for  the 
frequeney  of  the  estimates,  an  estimate  of  10  oeeurred  twiee  and  an  estimate  of  12 
oeeurred  twiee  and  Exeel  ehose  12  as  the  mode.  Again,  eonsensus  was  not  reaehed  but 
the  standard  deviation  was  minimized.  The  results  are  listed  in  Table  4.8,  shown  below. 
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Table  4.8:  Question  Two-  Average  Estimate  Results  By  Round 

Frequency 


Round 

Range 

(Member  1,2, 3, 4, 5) 

Mean 

Median 

Mode 

Std  Dev 

1 

5  to  20 

11.40 

10.00 

10.00 

5.46 

2 

8  to  15 

11.00 

10.00 

10.00 

2.65 

3 

8  to  15 

11.00 

10.00 

10.00 

2.65 

4 

8  to  12 

10.40 

10.00 

12.00 

1.67 

4,4,3  Question  Two-  High  Estimate 


The  high  estimate  followed  suit  with  the  other  two  estimates  by  starting  with  a 


wide  range  that  narrowed  as  the  rounds  oeeurred.  The  median  was  eonsistent  throughout 


ail  four  rounds  at  an  estimate  of  18  TDYs  and  this  time  the  mode  also  remained 


eonsistent  at  18  TDYs  as  well.  We  failed  again  to  reaeh  eonsensus,  but  the  standard 
deviation  was  minimized  to  the  greatest  extent  possible.  Of  note,  the  deviation  aetually 
inereased  by  two  hundredths  from  round  three  to  round  four.  The  inerease  in  standard 
deviation  was  due  to  member  five  dropping  their  estimate  from  20  TDYs  to  18  TDYs;  by 
doing  so  the  mean  deereased,  but  the  distance  between  the  mean  and  the  data  points 
increased.  The  results  for  the  question  two-high  estimate  are  found  below  in  Table  4.9. 


Table  4.9:  Question  Two-  High  Estimate  Results  By  Round 


Frequency 


Round 

Range 

(Member  1,2, 3, 4,5) 

Mean 

Median 

Mode 

Std  Dev 

1 

8  to  25 

I  8,18,18,25,25 

18.80 

18.00 

18.00 

6.98 

2 

12  to  25 

19.60 

18.00 

18.00 

5.50 

3 

16  to  25 

19.40 

18.00 

18.00 

3.44 

4 

16  to  25 

1  16,18,18,25,18 

19.00 

18.00 

18.00 

3.46 

4,5  Question  Three 


Estimate  how  many  people  normally  go  TDY  throughout  the  Milestone 
Decision  process. 

Question  three  established  another  portion  of  the  travel  estimate  in  our  cost  of 
oversight  algorithm.  The  goal  of  question  three  is  to  find  the  number  of  personnel  that 
actually  go  TDY  during  the  milestone  decision  process.  The  respondents  were  given  the 
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same  instructions  as  was  given  with  question  two  and  the  answers  will  be  presented  in  the 
similar  manner. 


4,5,1  Question  Three-  Low  Estimate 


Question  three  demonstrated  the  first  occurrence  of  the  inability  of  the  Delphi 
process  to  decrease  the  range  of  answers.  The  range  started  out  with  an  estimate  of  two 
to  eighty  people  going  TDY  in  support  of  the  milestone  decision  process,  went  up  in 
round  two  and  then  stabilized  back  at  the  original  round  one  estimate.  With  such  a  wide 
range  of  estimates,  the  mean  of  33.8  rounded  to  34  would  appear  to  be  the  most  reliable 
estimate  in  this  case.  The  modal  value  ended  up  at  an  estimate  of  80  people  going  TDY 
with  two  of  the  five  panelists  sticking  with  that  estimate.  Consensus  was  of  course  not 
reached  and  the  standard  deviation  ended  up  just  over  42. 


Table  4.10;  Question  Three-  Low  Estimate  Results  By  Round 


Frequency 


Round 

Range 

(Member  1,2, 3, 4, 5) 

Mean 

Median 

Mode 

Std  Dev 

1 

2  to  80 

60,80,2,2,3 

29.40 

3.00 

2.00 

37.73 

2 

2  to  120 

120,80,4,2,3 

41.80 

4.00 

#N/A 

54.98 

3 

2  to  80 

80,80,4,2,3 

33.80 

4.00 

80.00 

42.18 

4 

2  to  80 

80,80,4,2,3 

33.80 

4.00 

80.00 

42.18 

4,5,2  Question  Three-  Average  Estimate 

The  average  estimate  also  had  a  wide  range  of  values  although  unlike  the  low 
estimate,  this  time  the  mode  ended  up  at  a  low  value  of  ten  with  two  of  the  five  panelists 
going  with  that  estimate.  The  mean  ended  up  at  an  estimate  of  almost  63  people  going 
TDY.  Again,  the  mean  is  probably  the  safest  statistic  due  to  the  wide  range  of  estimates 
with  no  real  cluster  around  any  certain  estimate.  The  standard  deviation  actually 
increased  as  the  rounds  went  on  so  consensus  for  this  question  was  not  possible. 
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Table  4.11:  Question  Three-  Average  Estimate  By  Round 


Frequency 


Round 

Range 

(Member  1,2, 3, 4, 5) 

Mean 

Median 

Mode 

Std  Dev 

1 

4  to  120 

46.00 

10.00 

#N/A 

54.94 

2 

8  to  180 

1  180,120,8,10,10 

65.60 

10.00 

10.00 

79.92 

3 

10  to  160 

62.40 

12.00 

10.00 

72.24 

4 

10  to  160 

62.40 

12.00 

10.00 

72.24 

4,5,3  Question  Three-  High  Estimate 


The  high  estimate  results  listed  in  table  4.12  show  almost  the  same  pattern  that 


oeeurred  with  the  low  and  average  results.  There  was  a  wide  range  of  estimates  from  20 


to  200  in  the  final  round  and  the  mode  was  low  with  no  real  eluster  of  estimates  around 


any  one  value.  The  results  of  the  estimates  were  no  where  near  eonsensus  with  another 


very  large  standard  deviation. 


Table  4.12:  Question  Three-  High  Estimate  By  Round 


Frequency 


Round 

Range 

(Member  1,2, 3, 4,5) 

Mean 

Median 

Mode 

Std  Dev 

1 

6  to  200 

. . 

77.20 

50.00 

#N/A 

82.52 

2 

12  to  240 

100.40 

50.00 

#N/A 

103.25 

3 

20  to  200 

1  200,180,20,50,20 

94.00 

50.00 

20.00 

88.77 

4 

20  to  200 

94.00 

50.00 

20.00 

88.77 

4,6  Question  Four 


What  is  your  estimate  of  the  cost  for  each  person  on  each  TDY? 

Question  four  provides  the  final  number  for  the  travel  portion  of  the  eost  of 
oversight  formula.  By  multiplying  the  estimates  from  questions  two,  three,  and  four,  an 
estimate  for  the  eost  of  travel  in  the  oversight  proeess  ean  be  obtained.  Question  four 
will  provide  an  aetual  dollar  figure  estimate  for  the  eost  of  one  TDY  for  one  person  on  a 
team.  Results  are  presented  in  the  same  format  as  previous  questions. 
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4,6.1  Question  Four-  Low  Estimate 

Question  four’s  low  estimate  showed  a  pattern  of  a  eonstantly  decreasing  standard 
deviation.  The  range  of  estimates  however,  ensured  the  standard  deviation  would  remain 
high  and  that  consensus  could  not  be  met.  The  promising  news  from  these  estimates  is 
that  the  mean,  median  and  mode  were  very  close  as  all  were  at  or  near  an  estimate  of  one 
thousand  dollars  per  person  for  each  TDY.  The  results  can  be  seen  below  in  Table  4.13. 

Table  4.13:  Question  Four-  Low  Estimate  By  Round 


Frequency 


Round 

Range 

(Member  1,2, 3, 4, 5) 

Mean 

Median 

Mode 

Std  Dev 

1 

$450  to  $1200 

$450,$500,$1200,$1 000, $1000 

$830.00 

$1,000.00 

$1,000.00 

$334.66 

2 

$500  to  $1200 

$700, $500, $1 200, $1 000, $1 000 

$880.00 

$1,000.00 

$1,000.00 

$277.49 

3 

$600  to  $1200 

$700,$600,$1 200,$1 000,$1 000 

$900.00 

$1,000.00 

$1,000.00 

$244.95 

4 

$700  to  $1200 

$700,$900,$1 200,$1 000,$1 000 

$960.00 

$1,000.00 

$1,000.00 

$181.66 

4,6.2  Question  Four-  Average  Estimate 

As  with  the  low  estimates  for  question  four,  the  average  estimates  resulted  in  only 
a  moderate  range.  The  four  rounds  produced  a  result  no  where  near  consensus  however, 
with  a  large  standard  deviation  of  almost  three  hundred  and  five  dollars.  There  was  no 
modal  value,  but  the  mean  and  median  were  within  forty  dollars  of  each  other.  The 
results  can  be  seen  in  the  table  below. 


Table  4.14:  Question  Four-  Average  Estimate  By  Round 


Frequency 


Round 

Range 

(Member  1,2, 3, 4,5) 

Mean 

Median 

Mode 

Std  Dev 

1 

$900  to  $1600 

$1000,$900,$1600,$1000,$1500 

$1,200.00 

$1,000.00 

$1,000.00 

$324.04 

2 

$900  to  $1600 

$900,$1 000,$1 600,$1 000,$1 500 

$1,200.00 

$1,000.00 

$1,000.00 

$324.04 

3 

$900  to  $1600 

$900,$1200,$1600,$1000,$1500 

$1,240.00 

$1,200.00 

#N/A 

$304.96 

4 

$900  to  $1600 

$900,$1200,$1600,$1000,$1500 

$1,240.00 

$1,200.00 

#N/A 

$304.96 

4,6,3  Question  Four-  High  Estimate 

The  high  estimate  for  question  four  had  a  relatively  large  range,  seen  in  Table 
4.15,  but  the  range  remained  constant  throughout  all  four  rounds.  The  mode  and  median 
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were  the  same  for  the  last  two  rounds,  but  this  time  the  mean  was  a  bit  lower.  The 


standard  deviation  was  quite  large  and  eonsensus  was  not  met. 


Table  4.15:  Question  Four-  High  Estimate  By  Round 


Frequency 


Round _ Range _ (Member  1,2, 3, 4, 5) _ Mean  Median  Mode  Std  Dev 


1 

$1000  to  $2000 

$1400,$1200, $2000, $1000, $2000 

$1,520.00 

$1,400.00 

$2,000.00 

$460.43 

2 

$1000  to  $2000 

$1 200,$1 500,$2000,$1 000,$2000 

$1,540.00 

$1,500.00 

$2,000.00 

$456.07 

3 

$1000  to  $2000 

$1 200,$2000,$2000,$1 000,$2000 

$1,640.00 

$2,000.00 

$2,000.00 

$498.00 

4 

$1000  to  $2000 

$1 200,$2000,$2000,$1 000,$2000 

$1,640.00 

$2,000.00 

$2,000.00 

$498.00 

4.7  Travel  Cost  Estimate 


With  all  the  neeessary  data  eolleeted,  it  was  now  possible  to  develop  an  overall 
estimate  for  travel  eosts  using  the  previously  mentioned  algorithm.  To  review,  the  travel 
eost  estimate  will  be  developed  by  multiplying  the  estimates  from  question  two;  the 
number  of  TDYs  taken  by  one  person,  by  the  estimates  from  question  three;  the  total 
number  of  persons  who  go  TDY,  by  the  estimates  from  question  four;  the  eost  per  person 
for  eaeh  TDY.  The  results  ean  be  seen  below  in  Table  4.16. 


Table  4.16:  Estimates  of  Travel  Cost  for  One  Milestone 


Questions  2-4 


MEMBER  Travel-LOW  Travel-AVG  Travel-HIGH 


1 

$336,000.00 

$1,152,000.00 

$3,840,000.00 

2 

$576,000.00 

$1,728,000.00 

$6,480,000.00 

3 

$28,800.00 

$192,000.00 

$720,000.00 

4 

$12,000.00 

$100,000.00 

$1,250,000.00 

5 

$18,000.00 

$180,000.00 

$720,000.00 

MEAN 

$194,160.00 

$670,400.00 

$2,602,000.00 

STD  DEV 

$253,711.90 

$732,320.15 

$2,524,048.34 

The  estimates  were  ealeulated  by  member  for  low,  average  and  high  and  then  a 


mean  and  standard  deviation  was  ealeulated  for  all  five  members  for  the  low,  average. 


and  high  estimates.  Based  on  the  mean,  the  end  result  of  four  rounds  of  the  Delphi  was  a 


travel  estimate  that  ranged  from  about  $194  thousand  to  $2.6  million  for  one  milestone. 
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4.8  Question  Five 


Estimate  how  many  hours  are  spent  on  support  for  the  DAB  approval  process 
per  person,  not  including  TDY  travel  time,  but  actual  job  performance  while 
TDY  or  at  home  base.  (Slide  prep,  meeting  prep,  etc) 

In  question  five,  a  new  segment  of  the  eost  of  oversight  algorithm  is  started. 

Question  five  is  the  beginning  of  the  personnel  portion  of  the  estimate.  With  question 

five,  the  goal  is  to  find  the  number  of  hours  personnel  put  in  direetly  towards  the  DAB 


proeess. 

4,8,1  Question  Five-  Low  Estimate 


The  results,  shown  in  Table  4.17,  started  off  with  another  large  range.  The  range 


stabilized  by  round  two,  but  panelist  number  two  raised  their  estimate  for  the  number  of 


per  person  hours  spent  supporting  the  DAB  from  200  to  300  so  the  standard  deviation  did 


not  stabilize  until  round  three.  The  mode  was  almost  non  existent,  but  it  was  promising 


to  see  the  mean  and  the  median  so  elose  together,  hovering  at  about  400  hours. 


Consensus  was  not  met. 


Table  4.17:  Question  Five-  Low  Estimate  By  Round 


Frequency 


Round 

Range 

(Member  1,2, 3, 4, 5) 

Mean 

Median 

Mode 

Std  Dev 

1 

10  to  690 

372.00 

400.00 

#N/A 

287.18 

2 

200  to  690 

690,200,400,200,500 

398 

400 

200 

208.61 

3 

200  to  690 

418 

400 

#N/A 

188.73 

4 

200  to  690 

1 1 II 1 1 II 1 1 II 1 1 II 1 1 

418 

400 

#N/A 

188.73 

4,8,2  Question  Five-  Average  Estimate 


In  Table  4.18,  the  average  estimates  have  almost  the  same  pattern  as  the  low 


estimate  for  question  five.  This  time  however,  the  range  did  not  stabilize  until  the  third 


round,  but  the  standard  deviation  had  the  same  result  as  it  also  stabilized  by  the  third 
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round.  The  mean  and  median  had  relatively  elose  values  at  right  around  700  hours. 
There  was  no  mode  and  consensus  was  not  reached. 


Table  4.18:  Question  Five-  Average  Estimate  By  Round 


Frequency 


Round 

Range 

(Member  1,2, 3, 4, 5) 

Mean 

Median 

Mode 

Std  Dev 

1 

20  to  1040 

552.00 

660.00 

#N/A 

415.84 

2 

360  to  1040 

1  1040,360,660,600,700 

672 

660 

#N/A 

244.38 

3 

500  to  1040 

700 

660 

#N/A 

204.45 

4 

500  to  1040 

^11  III  II  II  II  II  Millie 

700 

660 

#N/A 

204.45 

4,8,3  Question  Five-  High  Estimate 

Table  4.19  below  shows  the  results  for  the  high  estimates.  Once  again  these 
showed  the  similar  pattern  as  the  earlier  portions  of  question  five.  The  range,  standard 
deviation,  mean  and  median  became  stable  by  the  third  round.  For  the  high  estimate, 
there  was  a  modal  value  of  one  thousand  hours.  The  mean,  median  and  modal  values 


were  all  close  at  around  one  thousand,  but  consensus  was  not  reached. 


Table  4.19:  Question  Five-  High  Estimate  By  Round 


Frequency 


Round 

Range 

(Member  1,2, 3, 4,5) 

Mean 

Median 

Mode 

Std  Dev 

1 

40  to  1 560 

780.00 

820.00 

#N/A 

569.21 

2 

520  to  1 560 

Hbl;lil.>iil:WillililillililiH 

980 

1000 

1000 

378.95 

3 

820  to  1 560 

Hbl;{iKel;{iI:WilblililbliliH 

1068 

1000 

1000 

284.82 

4 

820  to  1560 

Mil  1 1 H 1 1 1 1 1 II 1 1 II 1 1  M 

1068 

1000 

1000 

284.82 

4,9  Question  Six 


Estimate  how  many  people  are  normally  involved  with  the  preparation  process. 

Question  six  places  an  actual  number  of  personnel  into  the  second  portion  of  the 
algorithm  for  cost  of  oversight.  The  number  of  personnel  involved  in  the  preparation 
process  included  those  creating  slides,  preparing  briefings,  and  supporting  the  DAB.  The 
results  are  given  in  similar  format  as  previous  data  collected. 
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4,9.1  Question  Six-  Low  Estimate 


The  low  estimate,  seen  in  Table  4.20,  had  a  very  large  range  that  didn’t  get  mueh 
smaller  throughout  the  rounds.  No  panelists  changed  their  answers  in  the  third  or  fourth 
round  where  the  range  was  estimated  to  be  from  ten  to  one  hundred  and  fifty  people 
involved.  The  median  and  mode  were  the  same  for  rounds  two  through  four  with  an 
estimate  of  ten  personnel  involved.  With  such  a  great  range  and  large  standard  deviation, 
the  safest  statistic  would  appear  to  be  the  mean  with  a  value  of  fifty  two  however,  during 
the  last  two  rounds  three  of  the  five  panelists  agreed  to  an  estimate  of  ten  personnel  so  the 
median/mode  might  be  a  better  statistic  to  go  with. 


Table  4.20:  Question  Six-  Low  Estimate  By  Round 


Frequency 


Round 

Range 

(Member  1,2, 3, 4, 5) 

Mean 

Median 

Mode 

Std  Dev 

1 

1  to  200 

60,200,5,1,10 

55.20 

10.00 

#N/A 

84.40 

2 

5  to  150 

60,150,5,10,10 

47.00 

10.00 

10.00 

61.81 

3 

10  to  150 

80,150,10,10,10 

52.00 

10.00 

10.00 

62.61 

4 

10  to  150 

80,150,10,10,10 

52.00 

10.00 

10.00 

62.61 

4,9,2  Question  Six-Average  Estimate 


The  average  estimate,  located  in  Table  4.21,  showed  almost  identical  movement 


as  the  low  estimates.  No  estimates  were  changed  after  round  three  and  the  standard 


deviation  was  quite  large.  Consensus  was  not  reached. 


Table  4.21:  Question  Six-  Average  Estimate  By  Round 


Frequency 


Round 

Range 

(Member  1,2,3,4,5) 

Mean 

Median 

Mode 

Std  Dev 

1 

3  to  300 

90,300,8,3,25 

85.20 

25.00 

#N/A 

125.00 

2 

8  to  250 

90,250,8,25,25 

79.60 

25.00 

25.00 

100.29 

3 

25  to  225 

160,225,25,25,25 

92.00 

25.00 

25.00 

94.58 

4 

25  to  225 

160,225,25,25,25 

92.00 

25.00 

25.00 

94.58 
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4,9,3  Question  Six-  High  Estimate 


The  high  estimate,  seen  below  in  Table  4.22,  ended  up  with  the  exact  same  types 


of  statistics  as  the  low  and  average  estimates  did.  The  mode  ended  up  with  a  value  of 


sixty  personnel.  Consensus  was  not  reached. 


Table  4.22;  Question  Six-  High  Estimate  By  Round 


Frequency 


Round 

Range 

(Member  1,2, 3, 4, 5) 

Mean 

Median 

Mode 

Std  Dev 

1 

7  to  400 

120,400,12,7,60 

119.80 

60.00 

#N/A 

163.12 

2 

12  to  300 

120,300,12,60,60 

110.40 

60.00 

60.00 

112.70 

3 

60  to  275 

200,275,60,60,60 

131.00 

60.00 

60.00 

100.77 

4 

60  to  275 

200,275,60,60,60 

131.00 

60.00 

60.00 

100.77 

4,10  Question  Seven 


Estimate  the  cost  per  hour  for  each  person  involved  in  the  process. 

Question  Seven  provides  the  last  portion  of  the  personnel  cost  estimate  for  the 
total  cost  of  oversight.  With  the  estimates  provided  in  question  seven,  multiplied  by  the 
estimates  given  in  questions  five  and  six,  the  estimated  forecast  for  the  cost  of  personnel 
in  the  oversight  process  can  be  determined. 


4,10,1  Question  Seven-  Low  Estimate 


In  Table  4.23,  the  low  estimates  are  provided.  The  estimates  changed  in  the  first 


two  rounds  and  remained  unchanged  for  the  last  two  rounds.  The  standard  deviation  was 


reduced  and  leveled  out  after  round  three.  The  median  and  mode  remained  constant  after 


round  two.  In  the  final  round  the  mean,  median  and  mode  were  all  different,  but  were 
within  a  total  of  less  than  five  dollars. 
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Table  4.23:  Question  Seven-  Low  Estimate  By  Round 


Frequency 


Round 

Range 

(Member  1,2, 3, 4, 5) 

Mean 

Median 

Mode 

Std  Dev 

1 

$27.97  to  $60 

39.59 

32.00 

#N/A 

14.59 

2 

$32  to  $60 

$43, $50, $45.58, $60, $32 

46.12 

45.58 

#N/A 

10.21 

3 

$43  to  $60 

$43,$50,$45,58,$60,$43 

48.32 

45.58 

43.00 

7.13 

4 

$43  to  $60 

48.32 

45.58 

43.00 

7.13 

4.10.2  Question  Seven-  Average  Estimate 


Question  seven’s  average  estimates,  loeated  in  Table  4.24,  show  a  similar  pattern 


to  the  low  estimates.  The  estimates  stabilized  after  round  two,  but  the  standard  deviation 


doubled  as  did  the  total  differenee  between  the  mean,  median  and  mode.  Consensus  was 


not  met. 


Table  4.24:  Question  Seven-  Average  Estimate  By  Round 


Frequency 


Round 

Range 

(Member  1,2,3,4,5) 

Mean 

Median 

Mode 

Std  Dev 

1 

$31.26  to  $80 

$31.26,$65,$34,$80,$37 

49.45 

37.00 

#N/A 

21.79 

2 

$37  to  $80 

56.44 

54.20 

#N/A 

16.74 

3 

$46  to  $80 

58.24 

54.20 

46.00 

14.45 

4 

$46  to  $80 

1  $46,$65,$54.20,$80,$46 

58.24 

54.20 

46.00 

14.45 

4.10.3  Question  Seven-  High  Estimate 

The  high  estimate  for  question  seven,  seen  in  Table  4.25,  followed  suit  with  the 
low  and  average  estimates  as  the  standard  deviation  grew  eonsiderably,  doubling  from 
that  of  the  average  estimate  in  round  four.  The  mean,  median  and  mode  were  all  different 
with  a  total  differenee  of  almost  twenty  six.  The  high  mean  ean  be  aeeounted  for  by  the 
very  large  estimate  provided  by  panelist  number  four.  Due  to  the  skewed  mean,  the 
median  or  mode  might  provide  a  better  estimate  for  the  eost  per  hour  for  eaeh  person 
involved  in  the  proeess. 
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Table  4.25:  Question  Seven-  High  Estimate  By  Round 


Frequency 


Round 

Range 

(Member  1,2, 3, 4, 5) 

Mean 

Median 

Mode 

Std  Dev 

1 

$31.31  to  $120 

63.26 

42.00 

#N/A 

38.09 

2 

$42  to  $120 

71.61 

64.05 

#N/A 

31.86 

3 

$47  to  $120 

1  $47, $85, $64.05, $120, $47 

72.61 

64.05 

47.00 

30.76 

4 

$47  to  $120 

72.61 

64.05 

47.00 

30.76 

4.11  Personnel  Cost  Estimate 


With  all  the  neeessary  data  eolleeted,  it  was  now  possible  to  develop  an  overall 
estimate  for  personnel  eosts  using  the  previously  mentioned  algorithm.  The  personnel 
eost  estimate  was  developed  by  multiplying  the  estimates  from  question  five;  the  number 
of  hours  one  person  spends  in  support  of  a  milestone  review,  by  the  estimates  from 
question  six;  the  total  number  of  persons  who  support  a  milestone  review,  by  the 
estimates  from  question  seven;  the  eost  for  one  person  involved  in  the  milestone  review. 
The  results  ean  be  seen  below  in  Table  4.26. 


Table  4.26:  Estimates  of  Personnel  Cost  Eor  One  Milestone 


Questions  5-7 


MEMBER  Person-LOW  Person-AVG  Person-HIGH 


1 

$2,373,600.00 

$7,654,400.00 

$14,664,000.00 

2 

$2,250,000.00 

$7,312,500.00 

$22,440,000.00 

3 

$182,320.00 

$894,300.00 

$3,151,260.00 

4 

$120,000.00 

$1,200,000.00 

$7,200,000.00 

5 

$215,000.00 

$805,000.00 

$2,820,000.00 

MEAN 

$1,028,184.00 

$3,573,240.00 

$10,055,052.00 

STD  DEV 

$1,173,086.83 

$3,574,565.45 

$8,408,165.41 

The  estimates  were  ealeulated  by  member  for  low,  average  and  high  and  then  a 


mean  and  standard  deviation  was  ealeulated  for  all  five  members  for  the  low,  average. 


and  high  estimates.  Based  on  the  mean,  the  end  result  of  four  rounds  of  the  Delphi  was  a 


personnel  eost  estimate  that  ranges  from  just  over  $1  million  to  just  over  $10  million  for 


one  milestone. 


66 


4.12  Question  Eight 

Estimate  how  many  meetings  are  normally  held  from  the  PEO  preparation, 
through  DAB  approval.  (This  includes  meetings  while  TDY or  TDY prep 
meetings). 

Question  eight  provides  insight  on  how  meetings  are  included  into  the  oversight 
process.  By  multiplying  questions  eight,  nine,  and  ten,  we  will  get  an  idea  of  truly  what 
part  meetings  play  in  the  cost  of  oversight.  Question  eight  deals  specifically  with  the 
number  of  meetings  that  are  held  during  one  milestone  in  a  program.  The  results  are 
listed  in  the  following  three  paragraphs. 

4,12,1  Question  Eight-  Low  Estimate 

The  low  estimate,  in  Table  4.27,  shows  a  pretty  broad  range  that  solidified  by 
round  two  with  an  estimate  on  the  number  of  meetings  estimated  from  six  to  one  hundred 
twenty.  The  mean,  median,  and  mode  remained  constant  and  relatively  close  in  value 
after  the  third  round.  The  standard  deviation  actually  increased  from  round  one  to  round 
two,  driven  by  panelist  number  one’s  estimate,  but  then  went  down  and  stabilized  in 
rounds  three  and  four. 


Table  4.27;  Question  Eight-  Low  Estimate  By  Round 


Frequency 


Round 

Range 

(Member  1,2, 3, 4, 5) 

Mean 

Median 

Mode 

Std  Dev 

1 

5  to  50 

5,48,12,6,50 

24.20 

12.00 

#N/A 

22.81 

2 

6  to  120 

46.20 

45.00 

#N/A 

45.38 

3 

6  to  120 

53.40 

48.00 

48.00 

41.26 

4 

6  to  120 

53.40 

48.00 

48.00 

41.26 

4,12,2  Question  Eight-  Average  Estimate 


The  average  estimate,  seen  below  in  Table  4.28,  resulted  in  figures  that  performed 


similar  to  the  low  estimates.  The  range  remained  high  throughout  the  process,  but  the 
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mean,  median  and  mode  were  close.  This  time,  three  of  the  five  panelists  agreed  on  an 
average  estimate  of  60  meetings. 


Table  4.28:  Question  Eight-  Average  Estimate  By  Round 


Frequency 


Round 

Range 

(Member  1,2, 3, 4, 5) 

Mean 

Median 

Mode 

Std  Dev 

1 

8  to  100 

8,60,18,10,100 

39.20 

18.00 

#N/A 

40.01 

2 

10  to  180 

180,60,18,10,60 

65.60 

60.00 

60.00 

68.02 

3 

10  to  180 

180,60,60,10,60 

74.00 

60.00 

60.00 

63.09 

4 

10  to  180 

180,60,60,10,60 

74.00 

60.00 

60.00 

63.09 

4.12,3  Question  Eight-  High  Estimate 

The  high  estimates  for  question  eight  are  seen  in  Table  4.29.  Panelist  number  one 
went  from  an  initial  estimate  of  12  to  a  very  high  estimate  of  240  and  kept  that  estimate 
for  the  remaining  rounds.  The  high  estimate  is  again  skewing  the  mean  to  the  high  side. 
The  mode  and  median  were  constant  and  equal  in  both  rounds  three  and  four. 


Table  4.29:  Question  Eight-  High  Estimate  By  Round 


Frequency 


Round 

Range 

(Member  1,2,3,4,5) 

Mean 

Median 

Mode 

Std  Dev 

1 

12  to  150 

12,90,26,25,150 

60.60 

26.00 

#N/A 

58.49 

2 

25  to  240 

240,90,26,25,90 

94.20 

90.00 

90.00 

87.65 

3 

25  to  240 

240,90,90,25,90 

107.00 

90.00 

90.00 

79.50 

4 

25  to  240 

240,90,90,25,90 

107.00 

90.00 

90.00 

79.50 

4.13  Question  Nine 


What  do  you  estimate  as  the  length,  in  hours,  for  each  meeting? 

Question  nine  provides  a  length  for  each  meeting,  which  will  be  multiplied  by  the 
number  of  meetings  provided  in  question  eight  and  the  cost  per  hour  for  each  person 
attending,  which  will  be  provided  in  question  ten.  Question  nine  was  an  estimate  that 
came  in  with  relatively  low  standard  deviations  for  each  estimate,  but  overall,  was  not 
significantly  volatile  from  one  round  to  the  next.  Results  are  provided  in  the  following 
three  paragraphs. 
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4,13,1  Question  Nine-  Low  Estimate 

The  low  estimate  in  question  nine,  loeated  in  Table  4.30,  had  a  very  small  range 
with  an  estimate  of  two  to  four  hours  per  meeting  by  round  two.  No  answers  were 
changed  by  round  two  and  the  mean,  median  and  mode  ended  up  being  very  close  in 
value.  The  standard  deviation  closed  at  a  value  of  less  than  one,  but  consensus  was  not 
met. 


Table  4.30:  Question  Nine-  Low  Estimate  By  Round 


Frequency 


Round 

Range 

(Member  1,2, 3, 4, 5) 

Mean 

Median 

Mode 

Std  Dev 

1 

1  to  4 

2, 2, 2, 4,1 

2.20 

2.00 

2.00 

1.10 

2 

2  to  4 

2, 2, 2, 4, 2 

2.40 

2.00 

2.00 

0.89 

3 

2  to  4 

2, 2, 2, 4, 2 

2.40 

2.00 

2.00 

0.89 

4 

2  to  4 

2, 2, 2, 4, 2 

2.40 

2.00 

2.00 

0.89 

4,13,2  Question  Nine-  Average  Estimate 

The  average  estimate  for  question  nine,  listed  in  Table  4.31,  had  a  similar  pattern 
to  the  low  estimate.  Question  now  is  the  first  time  that  we  see  the  Delphi  process  really 
starting  to  work  as  there  is  a  4  out  of  5  agreement  by  round  two  that  holds  throughout. 
This  time,  the  mean  is  skewed  a  bit  high  by  panelist  number  four’s  high  estimate  of 
sixteen,  so  the  mode  of  four  hours  per  meeting  may  be  a  better  estimate  to  go  with. 


Table  4.3 1 :  Question  Nine-  Average  Estimate  By  Round 


Frequency 


Round 

Range 

(Member  1,2,3,4,5) 

Mean 

Median 

Mode 

Std  Dev 

1 

1.5  to  16 

4,4,4,16,1.5 

5.90 

4.00 

4.00 

5.75 

2 

4  to  16 

4,4,4,16,4 

6.40 

4.00 

4.00 

5.37 

3 

4  to  16 

4,4,4,16,4 

6.40 

4.00 

4.00 

5.37 

4 

4  to  16 

4,4,4,16,4 

6.40 

4.00 

4.00 

5.37 

4,13,3  Question  Nine-  High  Estimate 


The  high  estimate  for  question  nine,  located  in  Table  4.32  below,  had  range 


performance  similar  to  the  low  and  average  estimates  where  it  stayed  the  same  through 
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rounds  two  through  four;  however,  the  high  estimate  was  the  furthest  from  eonsensus. 
All  estimates  stayed  the  same  for  all  members  by  round  two  only  this  time  the  mode  is 
basieally  split  with  two  of  the  five  panelists  going  with  a  high  estimate  of  8  hours  per 
meeting,  two  of  the  five  panelists  going  with  12  hours  per  meeting  and  panelist  number 
four  remaining  eonsistent  with  a  high  estimate. 


Table  4.32:  Question  Nine-  High  Estimate  By  Round 


Frequency 


Round 

Range 

(Member  1,2, 3, 4, 5) 

Mean 

Median 

Mode 

Std  Dev 

1 

2  to  24 

12,8,12,24,8 

10.80 

8.00 

8.00 

8.20 

2 

8  to  24 

12,8,12,24,8 

12.80 

12.00 

12.00 

6.57 

3 

8  to  24 

12.80 

12.00 

12.00 

6.57 

4 

8  to  24 

12,8,12,24,8 

12.80 

12.00 

12.00 

6.57 

4.14  Question  Ten 


What  is  the  cost  per  hour  of  each  person  involved  in  the  meetings? 

Question  ten  provided  the  eost  per  person  to  include  in  the  final  portion  of  the 
cost  of  oversight  estimate  for  meetings  conducted  for  a  milestone.  The  results  of 
question  ten  mirrored  the  results  of  question  seven,  due  to  both  dealing  with  the  cost  of 
personnel  per  hour.  The  results  will  still  be  provided  separately  due  to  future  discussion 
on  the  cost  of  oversight  and  the  analysis  portion  of  the  thesis.  The  estimates  will  be 
provided  in  the  same  format  as  previous  questions. 


4,14,1  Question  Ten-  Low  Estimate 


Question  ten’s  low  estimate,  listed  in  Table  4.33,  ended  with  a  relatively  low 
standard  deviation  despite  beginning  with  a  pretty  broad  range.  All  members  locked  into 
their  estimates  by  round  three  which  produced  a  median  and  mode  that  were  close  in 
value  however  the  mean  was  a  bit  higher  driven  by  two  of  the  five  panelists  estimating 
the  cost  per  hour  of  each  member  at  a  meeting  of  $60  per  hour.  The  mode  is  actually 
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split  with  two  of  the  five  panelists  going  with  an  estimate  of  $43  per  hour.  Panelist  three 


went  with  a  very  preeise  estimate  of  $45.58  per  hour  putting  three  of  the  five  panelists  in 


the  same  ballpark.  Based  on  this  fact,  the  mode  might  be  the  safest  statistic  to  use  for  the 


estimate  of  the  cost  per  hour  of  the  people  at  milestone  meetings. 


Table  4.33:  Question  Ten-  Low  Estimate  By  Round 


Frequency 


Round 

Range 

(Member  1,2, 3, 4, 5) 

Mean 

Median 

Mode 

Std  Dev 

1 

$27.97  to  $60 

42.79 

34.00 

60.00 

15.86 

2 

$32  to  $60 

48.12 

45.58 

60.00 

11.99 

3 

$43  to  $60 

1  $43, $60, $45.58, $60, $43 

50.32 

45.58 

43.00 

8.90 

4 

$43  to  $60 

50.32 

45.58 

43.00 

8.90 

4.14.2  Question  Ten-  Average  Estimate 

The  average  estimate,  seen  in  Table  4.34,  initially  had  a  large  range  of  almost  50 
and  a  relatively  large  standard  deviation  which  was  almost  half  of  the  range.  Similar  to 
the  action  with  the  low  estimates,  with  the  average  estimates,  panelists  locked  into  their 
values  by  the  third  round.  This  time  however,  the  mean,  median,  and  mode  weren’t  real 
close  in  value.  Two  of  the  five  panelists  did  end  up  agreeing  to  a  value  of  $46  per  hour, 
however,  the  remaining  panelists  all  went  with  higher  values.  The  mean  is  skewed  high 
by  panelist  four’s  estimate  of  $80  per  hour,  but  with  most  of  the  panel  giving  estimates 
higher  than  the  mode,  the  mean  is  probably  the  best  estimate  in  this  case. 


Table  4.34:  Question  Ten-  Average  Estimate  By  Round 


Frequency 


Round 

Range 

(Member  1,2,3,4,5) 

Mean 

Median 

Mode 

Std  Dev 

1 

$31.26  to  $80 

52.25 

38.00 

#N/A 

23.26 

2 

$37  to  $80 

58.44 

54.20 

#N/A 

18.52 

3 

$46  to  $80 

$46,$75,$54.20,$80,$46 

60.24 

54.20 

46.00 

16.20 

4 

$46  to  $80 

$46,$75,$54.20,$80,$46 

60.24 

54.20 

46.00 

16.20 
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4,14,3  Question  Ten-  High  Estimate 

The  high  estimate,  listed  in  Table  4.35,  produeed  figures  that  performed  in  a 
similar  manner  to  the  low  and  average  estimates.  Panelists  again  loeked  into  their 
estimates  by  round  three  and  similar  to  the  average  estimates,  only  two  of  the  five 
panelists  agreed  on  an  estimate  in  the  end  with  the  remaining  panelists  going  higher.  For 
the  same  reason  as  that  given  in  the  seetion  on  the  average  estimate,  the  mean  is  probably 
the  best  estimate  in  this  case. 


Table  4.35:  Question  Ten-  High  Estimate  By  Round 


Frequency 


Round 

Range 

(Member  1,2, 3, 4, 5) 

Mean 

Median 

Mode 

Std  Dev 

1 

$31.31  to  $120 

66.86 

46.00 

#N/A 

38.51 

2 

$42  to  $120 

73.61 

64.05 

#N/A 

33.20 

3 

$47  to  $120 

1  $47, $95, $64.05, $120, $47 

74.61 

64.05 

47.00 

32.06 

4 

$47  to  $120 

74.61 

64.05 

47.00 

32.06 

4,15  Meeting  Cost  Estimate 

With  all  the  necessary  data  collected,  it  was  now  almost  possible  to  develop  an 
overall  estimate  for  the  cost  of  all  the  meetings  supporting  the  DAB  milestone  review 
process  using  the  previously  mentioned  algorithm.  The  meeting  cost  estimate  will  be 
developed  by  multiplying  the  estimates  from  question  eight;  the  number  of  meetings  held 
from  PEO  preparation  to  DAB  approval,  by  the  estimates  from  question  nine;  the  length 
in  hours  of  each  meeting,  by  the  estimates  from  question  ten;  the  cost  for  one  person 
involved  in  one  meeting.  Using  the  algorithm,  we  are  able  to  develop  a  cost  estimate  for 
one  person  at  all  of  the  meetings  necessary  to  get  through  milestone  approval.  In  order  to 
come  up  with  an  estimate  for  the  cost  of  all  the  meetings,  we  polled  the  Delphi  panel  and 
asked  them  to  provide  a  low,  medium  and  high  estimate  for  the  number  of  people  who 
attend  the  meetings.  The  results  can  be  seen  below  in  Table  4.36. 
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Table  4.36:  Estimates  of  the  Number  of  People  who  Attend  Meetings 


Member# _ Low _ Med _ High 


1 

5 

12 

25 

2 

6 

15 

40 

3 

10 

20 

30 

4 

5 

10 

25 

5 

4 

8 

12 

Mean 

6.00 

13.00 

26.40 

Standard  Dev 

2.35 

4.69 

10.11 

With  all  of  the  necessary  figures  now  available,  the  data  was  compiled  and  produced  the 


following  estimates  for  the  cost  of  the  meetings  in  support  of  one  milestone. 


Table  4.37:  Estimates  of  Meeting  Costs  for  One  Milestone 


Questions  8-11 


MEMBER _ Meeting-LOW  Meeting-AVG  Meeting-HiGH 


1 

$51,600.00 

$397,440.00 

$3,384,000.00 

2 

$34,560.00 

$270,000.00 

$2,736,000.00 

3 

$43,756.80 

$260,160.00 

$2,075,220.00 

4 

$7,200.00 

$128,000.00 

$1,800,000.00 

5 

$15,480.00 

$88,320.00 

$406,080.00 

MEAN 

$30,519.36 

$228,784.00 

$2,080,260.00 

STD  DEV 

$18,747.83 

$123,506.69 

$1,119,283.55 

The  estimates  were  calculated  by  panel  member  on  their  low,  average  and  high 


responses  and  then  a  mean  and  standard  deviation  was  calculated.  Based  on  the  mean. 


the  end  result  of  four  rounds  of  the  Delphi  was  a  cost  of  meetings  estimate  that  ranged 


from  about  $30.5  thousand  almost  $2.1  million  for  one  milestone. 

4.16  Summary  of  Results 

Estimates  for  the  three  major  oversight  cost  portions  of  a  milestone  review  for  an 


“inside  the  box”  MDAP  were  developed  using  a  simple  algorithm.  By  adding  these 


individual  estimates  together,  we  can  arrive  at  an  overall  estimate  for  the  cost  of  one 


milestone  decision  point.  The  results  of  summing  the  estimates  for  travel  cost,  personnel 


cost,  and  meeting  cost  can  be  seen  below  in  Table  4.38 
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Table  4.38:  Estimates  for  Oversight  Cost  of  One  Milestone 


Milestone  Decision  Point  (MDP) 


MEMBER _ MDP  Low _ MDP  Avg _ MDP  High 


1 

$2,761,200.00 

$9,203,840.00 

$21,888,000.00 

2 

$2,860,560.00 

$9,310,500.00 

$31,656,000.00 

3 

$254,876.80 

$1,346,460.00 

$5,946,480.00 

4 

$139,200.00 

$1,428,000.00 

$10,250,000.00 

5 

$248,480.00 

$1,073,320.00 

$3,946,080.00 

MEAN 

$1,252,863.36 

$4,472,424.00 

$14,737,312.00 

STD  DEV 

$1,423,444.51 

$4,369,992.94 

$11,739,247.49 

The  three  groups  of  estimates  for  travel,  personnel,  and  meeting  eosts  were  added 


together  to  develop  an  estimated  milestone  eost.  The  mean  statistie  shows  a  milestone 
oversight  eost  range  from  almost  $1.3  million  to  nearly  $14.8  million.  By  itself,  this 


statistie  is  interesting  enough  however  more  value  ean  be  added  by  eomparing  this  range 


to  the  range  for  oversight  eosts  developed  by  Col  Caldwell’s  ORPAT  team.  Reeall  that 


they  eame  up  with  an  average  estimate  of  $10-12  million  for  a  single  milestone  and  an 


estimate  of  $40-50  million  for  an  entire  joint  aequisition  program  in  1994  dollars  (14:9). 


Using  raw  inflation  indiees,  the  ORPAT  team’s  figures  ean  be  inflated  to  fiseal  year  2003 


dollars  or  the  figures  we  developed  eould  be  brought  baek  to  1994  dollars.  We  ehose  to 


ealeulate  the  latter  and  the  results  ean  be  seen  below  in  Table  4.39. 


Table  4.39:  FY2003  Milestone  Oversight  Costs  in  FY1994  Dollars 

Milestone  Decision  Point  (MDP)-Adjusted 


Milestone 

MDP  Low 

MDP  Avg 

MDP  High 

MEAN 

$1,252,863.36 

$4,472,424.00 

$14,737,312.00 

3080  Raw  Indice  (1994) 

0.885 

0.885 

0.885 

Adjusted  Mean 

$1,108,784.07 

$3,958,095.24 

$13,042,521.12 

As  the  ehart  shows,  the  “high”  milestone  estimate  falls  elose  to  the  ORPAT 


estimate  when  the  dollars  are  the  same.  The  proeurement  or  3080  index  was  used 
beeause  we’re  dealing  with  the  proeurement  of  weapons  systems.  If  the  index  for  wages 
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was  used,  the  figure  would  be  smaller  since  the  military  wage  index  is  .750  and  the 
civilian  employee  index  is  .738  (41:1). 

After  developing  a  milestone  oversight  cost  estimate,  a  total  program  oversight 
cost  estimate  is  arrived  at  quite  easily.  Our  methodology  for  developing  a  total  program 
oversight  cost  was  simply  to  multiply  the  milestone  figures  by  three  to  represent  the  three 
milestones.  The  assumption  here  is  that  each  milestone  costs  relatively  the  same.  We 
decided  to  stick  to  that  assumption  because  a  whole  series  of  research  could  be  conducted 
on  the  cost  difference  from  milestone  to  milestone.  The  results  of  our  calculation  for 


total  program  cost  can  be  seen  in  Table  4.40  below. 


Table  4.40:  Estimates  for  Total  Program  Oversight  Costs 


MEMBER  Program  Low  Program  Avg  Program  High 


1 

$8,283,600.00 

$27,611,520.00 

$65,664,000.00 

2 

$8,581,680.00 

$27,931,500.00 

$94,968,000.00 

3 

$764,630.40 

$4,039,380.00 

$17,839,440.00 

4 

$417,600.00 

$4,284,000.00 

$30,750,000.00 

5 

$745,440.00 

$3,219,960.00 

$11,838,240.00 

MEAN 

$3,758,590.08 

$13,417,272.00 

$44,211,936.00 

STD  DEV 

$4,270,333.53 

$13,109,978.82 

$35,217,742.48 

Looking  at  the  range  on  the  mean  we’ve  answered  research  question  number  one 
as  we  see  a  total  program  oversight  cost  estimate  from  almost  $3.8  million  to  over  $44.2 


million.  Again,  it  is  interesting  to  compare  this  range  to  the  range  developed  by  the 


ORPAT  team  of  $40-$50  million.  To  make  the  comparison  meaningful  we  again  brought 


our  figures  back  to  1994  dollars  and  the  results  can  be  seen  in  Table  4.41  below. 


Table  4.41:  FY2003  Total  Program  Oversight  Costs  in  FY1994  Dollars 


Program 

Program  Low 

Program  Avg 

Program  High 

MEAN 

$3,758,590.08 

$13,417,272.00 

$44,211,936.00 

3080  Raw  Indice  (1994) 

0.885 

0.885 

0.885 

Adjusted  Mean 

$3,326,352.22 

$11,874,285.72 

$39,127,563.36 
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Our  “high”  program  oversight  cost  estimate  of  just  over  $39.1  million  fell  a  little 
below  the  estimate  of  $40-$50  million  developed  by  the  ORPAT  team  this  time.  The 
ORPAT  estimate  however,  may  be  on  the  high  side  since  back  when  it  was  developed 
“box”  acquisition  programs  went  through  five  versus  today’s  three  milestones.  In  fact,  it 
was  Col  Caldwell’s  ORPAT  that  recommended  reducing  the  number  of  milestones.  It 
may  be  that  the  team’s  recommendation  is  resulting  in  actual  benefit  to  today’s  defense 
acquisition. 

The  goal  of  the  Delphi  Method  was  to  complete  at  least  four  rounds  while  trying 
to  reach  consensus.  The  objectives  were  clear  for  how  consensus  would  be  determined. 
The  rule  was  met  for  all  ten  questions  provided  in  the  survey  and  all  objectives  for  the 
data  collection  portion  were  met. 

Now  that  the  estimates  have  been  provided,  the  information  will  be  placed  in 
statistical  software  as  a  database.  Each  respondent  will  have  their  estimates  entered  for 
each  question.  This  will  be  compared  with  other  respondents  from  the  theses  research 
conducted  by  Neal  and  DeReus.  When  comparing  all  of  the  estimates  together,  an 
analysis  of  variance  test  will  be  conducted  by  question,  by  type  of  regulatory  guidance 
policy  programs  typically  fall  under,  (i.e..  Space,  DoDD  5000  series,  or  Virtual 
oversight).  Once  this  analysis  has  been  completed,  the  results  and  analysis  will  be 
presented  in  chapter  five  to  see  if  there  truly  is  a  difference  in  the  cost  of  oversight  among 
programs. 
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5,0  Analysis 


5.1  Overview 

The  goal  of  chapter  5  is  to  compare  the  results  from  the  final  round  of  the  Delphi 
surveys  for  each  of  the  DoD  acquisition  disciplines  examined.  The  first  section  will 
contain  a  question  by  question  breakdown  comparing  the  responses  from  the  DoD  5000 
or  “box”  program  surveys  to  the  C3I  or  “virtual  box”  program  surveys  to  the  Space  or 
“outside  the  box”  program  surveys.  Since  question  one  dealt  with  identifying  cost  drivers 
and  left  little  basis  for  a  statistical  comparison,  there  will  be  a  qualitative  comparison  of 
that  question.  Questions  two  through  ten  will  be  quantitatively  compared  through 
hypothesis  testing  and  use  of  probability  or  p-values.  The  stated  null  hypothesis  is  that 
there  is  no  statistical  difference  between  the  population  means  for  each  of  the  disciplines. 
The  common  p-value  of  .05  will  be  used  to  test  this  null  hypothesis.  If  in  comparing 
disciplines,  a  p-value  of  less  than  .05  results,  then  the  null  hypothesis  will  be  rejected  and 
we’ll  conclude  that  there  is  a  statistically  significant  difference  between  the  population 
means  of  the  disciplines.  A  p-value  of  greater  than  or  equal  to  the  .05  significance  level 
will  force  us  to  fail  to  reject  the  null  hypothesis  that  there  is  no  difference  between  the 
population  means.  Failing  to  reject  the  null  lends  to  the  conclusion  that  the  differences 
that  we  identified  were  the  result  of  error  in  our  random  sampling  of  the  populations  of 
“box”,  “virtual  box”  and  “outside  the  box”  programs,  not  from  true  difference  between 
the  population  means.  The  second  section  will  qualitatively  compare  the  total  program 
costs  deduced  from  chapter  four.  The  final  section  of  this  chapter  will  summarize  the 
chapter  five  results  and  examine  potential  areas  for  future  research  of  this  topic. 
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5.2  Question  One 

Stated  from  the  survey,  1.  From  the  Program  Executive  Officer  (PEO)  request 
for  a  Defense  Acquisition  Board  (DAB  to  the  DAB  milestone  approval,  what  are 
the  five  major  cost  drivers  in  the  oversight  process? 

The  goal  of  question  one  was  to  obtain  the  five  key  cost  drivers  that  the 

respondents  felt  drove  the  cost  of  oversight.  In  the  table  below,  the  top  drivers  identified 

from  each  of  the  disciplines  are  listed.  At  first  glance  it  is  interesting  to  note  that  there 

are  different  drivers  on  each  of  the  lists — no  driver  from  the  “box”  list  appeared  on  the 

space  list  or  C3I  list  and  so  on.  The  drivers  on  the  DoDD  5000  list  were  very  top  level 

dealing  with  issues  like  spiral  development,  how  many  services  were  involved  etc.  The 

space  list  focused  on  drivers  that  were  much  more  “ground”  level  like  actual  TDY  costs 

involved.  The  C3I  list  of  drivers  was  most  like  the  DoDD  5000  list  in  that  again  the 

focus  was  more  top  level  dealing  with  topics  like  requirements  definition  and 

technologies.  Though  the  drivers  were  different  in  all  three  lists,  that  in  no  way  supports 

a  conclusion  that  the  three  disciplines  all  face  different  oversight  cost  drivers.  For  this 

first  generation  of  this  thesis,  a  Delphi  was  conducted  using  five  member  panels.  If 

perhaps  larger  panels  were  employed  or  if  panelists  with  different  backgrounds  were  used 

the  results  might  have  been  different.  Additionally,  question  one  was  left  very  open 

ended,  allowing  the  panelists  to  brainstorm  cost  drivers  from  any  level.  If  perhaps  in 

future  generations  of  this  thesis  we  provided  a  baseline  list  of  drivers  at  the  specific  area 

of  oversight  we  wanted  to  focus  on,  we  might  have  had  a  better  chance  at  getting  similar 

oversight  cost  drivers  on  the  lists.  One  possibility  for  the  fact  that  the  drivers  identified 

for  both  the  DoD  oversight  process  and  its  virtual  cousin  were  very  top  level  and  the 

drivers  for  the  space  oversight  process  dealt  with  more  ground-level  issues  is  that  fact 
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that  the  DoD  and  C3I  processes  are  governed  by  a  centralized  body  of  regulations — the 
DoD  5000  series.  The  very  reason  the  space  oversight  process  was  allowed  to  operate 
outside  the  box  was  to  allow  for  flexibility  in  the  management  of  the  process  in  the  hopes 
of  achieving  greater  efficiency.  Because  space  oversight  is  more  decentralized,  the  cost 
drivers  may  now  come  from  a  factors  closer  to  the  process. 

Table  5.1:  Question  One-Oversight  Cost  Drivers 


Drivers  Picked-DoDD  5000 

Rank 

Program  is  Multi-Service  |  | 

1 

Whether  Completely  new  system  or  just  block  upgrade 

2 

Number  of  Technologies  going  into  the  system 

3 

Number  of  Systems  the  System  must  interact  with 

4 

1  Milestone  B  (requires  most  documents;  30  to  be  generated  for  review) 

5 

Drivers  Picked-Soace 

Rank 

|Time  away  from  primary  responsibilities  while  supporting  IPA  at  expense  of  rest  of  program 

1 

TDY  from  the  program  office  to  IPA  or  IPA  folks  to  program  office 

2 

IPA  Personnel  Costs  (Program  Evaluation)  |  | 

3 

Salaries  of  IPA  core  members  and  "gray  beard"  members  who  are  not  government  employees 

4 

IPA  Travel/PerDiem  costs  (Team  and  support  personnel)  | 

5 

Drivers  Picked-C3l 

Rank 

Lack  of  functional  reguirements  that  are  clearly  defined  and  understood 

1 

Negotiating  viewpoints  of  the  various  stakeholders... acg  strategy  re-do 

2 

The  serial  process  of  document  approval  by  the  several  echelons  of  oversight 

3 

The  lack  of  established  architectures  and  the  resulting  need  for  unigue  C4ISP  efforts 

4 

Changing  oversight  reguirements;the  way  we  did  things  previously  not  work  now  due  to 

5 

changing  personalities,  policy  etc. -reguires  climbing  the  learning  curve  again 

5.3  Question  Two 

From  the  PEO  recommendation,  to  the  DAB  approval  of  the  milestone,  use 
your  professional  judgment  and  estimate  how  many  TDYs  are  taken  by  one 
person  to  get  one  program  through  one  Milestone. 

The  goal  of  question  two  was  to  find  out  how  many  TDYs  are  taken  by  one 

individual  in  one  program  to  get  through  one  milestone.  In  the  table  below,  the  p-values 

for  the  comparisons  between  DoD  5000  and  Space  and  DoD  5000  and  C3I  are  listed  for 

low,  average,  and  high  range  responses.  Shaded  p-values  indicate  cases  where  we  failed 

to  reject  the  null  hypothesis  that  there  is  no  significant  difference  between  the  population 
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means  for  the  disciplines.  The  table  immediately  shows  that  in  almost  all  cases,  there 
was  a  statistically  significant  difference  between  the  population  means  of  the  disciplines 
for  the  question  of  how  many  TDY’s  are  taken.  The  only  case  where  we  failed  to  reject 
the  null  hypothesis  was  in  comparing  the  high  value  responses  between  “box”  programs 
and  C3I  programs.  Here  the  p-value  was  almost  .18;  a  value  far  greater  than  the  .05 
significance  level  we  established.  Looking  at  the  “box”  responses  the  range  at  the  high 
level  was  from  16  to  25  with  a  mean  of  19.  The  C3I  responses  ranged  from  12  to  30  with 
a  mean  of  25.5. 


Table  5.2;  Question  Two-Comparison  of  p-Values 
Question  2  p-Values  (.05  significance  ievei) 


COMPARISON 

LOW 

AVG 

HIGH 

Space  vs  5000 

0.0001 

0.0005 

0.0004 

C3I  vs  5000 

0.008 

0.0317 

0.1766 

Nuii  Hypothesis:  There  is  no  significant  difference 

Faii  to  reject:  No  difference  (Higiighted  in  Grey)-P>=.05 

Reject:  There  is  a  significant  difference  (P<.05) 

5.4  Question  Three 

Estimate  how  many  people  normally  go  TDY  throughout  the  Milestone 

Decision  process. 

Question  three  established  another  portion  of  the  travel  estimate  in  our  cost  of 
oversight  algorithm.  The  goal  of  question  three  was  to  find  the  number  of  personnel  that 
actually  go  TDY  during  the  milestone  decision  process.  As  seen  in  the  table  below,  all 
comparisons  resulted  in  very  high  p-values  so  in  all  cases  we  failed  to  reject  the  null 
hypothesis.  Failing  to  reject  the  null  does  not  conclusively  mean  that  that  the  population 
means  for  the  question  of  the  number  of  people  that  go  TDY  are  the  same  however,  based 
on  the  data  we  gathered  we  can  not  disprove  that  conclusion.  In  analyzing  the  responses 
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from  each  of  the  disciplines,  DoD  5000  responses  ranged  from  20  to  200  with  a  mean  of 
94,  the  space  responses  ranged  from  30  to  45  with  a  mean  of  35  and  the  C3I  responses 
ranged  from  12  to  40  with  a  mean  of  23.  In  both  comparisons  though  the  mean  from  the 
DoD  responses  far  exceeded  the  means  from  space  and  C3I,  their  means  did  in  fact  fall 
within  the  DoD  range. 


Table  5.3:  Question  Three-Comparison  of  p-Values 
Question  3  p-Values  (.05  significance  ievei) 


COMPARISON 

LOW 

AVG 

HIGH 

Space  vs  5000 

0.4934 

0.3418 

0.2323 

031  vs  5000 

0.2769 

0.2335 

0.1612 

Nuii  Hypothesis:  There  is  no  significant  difference 

Faii  to  reject:  No  difference  (Higiighted  in  Grey)-P>=.05 

Reject:  There  is  a  significant  difference  (P<.05) 

5.5  Question  Four 

What  is  your  estimate  of  the  cost  for  each  person  on  each  TDY? 

Question  four  provided  the  final  number  for  the  travel  portion  of  the  cost  of 
oversight  formula.  By  multiplying  the  estimates  from  questions  two,  three,  and  four,  an 
estimate  for  the  cost  of  travel  in  the  oversight  process  can  be  obtained.  Question  four 
provided  an  actual  dollar  figure  estimate  for  the  cost  of  one  TDY  for  one  person  on  a 
team.  Based  on  the  p-values  seen  below,  we  failed  to  reject  the  null  hypothesis  in  only 
two  of  the  six  possible  cases.  The  first  case  of  failing  to  reject  was  on  the  comparison  of 
average  responses  between  DoD  5000  and  space.  The  DoD  responses  for  the  average 
cost  of  each  TDY  ranged  from  $900  to  $1,600  with  a  mean  of  $1,240  and  the  space 
responses  ranged  from  $1,200  to  $1,800  with  a  mean  of  $1,450 — in  both  cases  the  means 
from  one  set  of  responses  fell  into  the  range  of  the  other.  The  second  case  of  failing  to 
reject  was  in  comparing  low  estimates  for  DoD  5000  to  C3I  estimates.  For  DoD,  the  low 
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estimate  for  the  eost  of  eaeh  TDY  ranged  from  $700  to  $1,200  with  a  mean  of  $960  and 


the  C3I  responses  ranged  from  $1,000  to  $1,200  with  a  mean  of  $1,050.  Here,  the  DoD 


mean  did  not  fall  within  the  C3I  range,  but  the  C31  mean  did  fall  within  the  range  of  low 
DoD  5000  responses. 

Table  5.4:  Question  Four-Comparison  of  p-Values 
Question  4  p-Values  (.05  significance  ievei) 


COMPARISON 

LOW 

AVG 

HIGH 

Space  vs  5000 

0.043 

0.3136 

0.0087 

031  vs  5000 

0.4071 

0.0123 

0.0017 

Nuii  Hypothesis:  There  is  no  significant  difference 

Faii  to  reject:  No  difference  (Higiighted  in  Grey)-P>=.05 

Reject:  There  is  a  significant  difference  (P<.05) 

5.6  Travel  Computation 

The  table  below  displays  the  estimates  for  the  eost  of  travel  assoeiated  with  the 
different  oversight  proeesses.  Although  in  the  earlier  eomparisons  of  travel  estimates, 
spaee  oversight  was  higher  in  eost  per  TDY,  the  overall  range  of  its  total  eosts  are  lower. 
This  again  is  hypothesized  that  due  to  the  IPA  proeess,  the  eost  of  eaeh  TDY  is  inereased, 
but  the  total  number  of  TDYs  are  deereased,  resulting  in  an  overall  deerease  in  total  TDY 
eosts.  This  seems  to  hold  true  when  aetually  eomparing  the  total  eost  for  travel  among  all 
three  oversight  proeesses. 
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Table  5.5:  Travel  Cost  Estimates  for  Each  Oversight  Process 


5000  -  T ravel 

MEMBER 

Travel -LOW 

Travel-AVG 

Travel-HIGH 

1 

$336,000.00 

$1,152,000.00 

$3,840,000.00 

2 

$576,000.00 

$1,728,000.00 

$6,480,000.00 

3 

$28,800.00 

$192,000.00 

$720,000.00 

4 

$12,000.00 

$100,000.00 

$1,250,000.00 

5 

$18,000.00 

$180,000.00 

$720,000.00 

MEAN 

$194,160.00 

$670,400.00 

$2,602,000.00 

STD  DEV 

$253,711.90 

$732,320.15 

$2,524,048.34 

Space 

MEMBER 

Travel-LOW 

Travel-AVG 

Travel-HIGH 

1 

$9,750.00 

$144,000.00 

$648,000.00 

2 

$30,000.00 

$195,000.00 

$913,500.00 

3 

$10,875.00 

$150,000.00 

$630,000.00 

4 

$35,000.00 

$144,000.00 

$792,000.00 

MEAN 

$21,406.25 

$158,250.00 

$745,875.00 

STD  DEV 

$12,979.70 

$24,662.72 

$133,206.84 

C3I  -  Travel 

MEMBER 

Travel-LOW 

Travel-AVG 

Travel-HIGH 

1 

$36,000.00 

$400,000.00 

$3,000,000.00 

2 

$42,000.00 

$94,500.00 

$432,000.00 

3 

$60,000.00 

$400,000.00 

$2,100,000.00 

4 

$240,000.00 

$1,200,000.00 

$4,800,000.00 

MEAN 

$94,500.00 

$523,625.00 

$2,583,000.00 

STD  DEV 

$97,534.61 

$473,355.99 

$1,821,086.49 

5.7  Question  Five 

Estimate  how  many  hours  are  spent  on  support  for  the  DAB  approval  process 
per  person,  not  including  TDY  travel  time,  but  actual  job  performance  while 
TDY  or  at  home  base.  (Slide  prep,  meeting  prep,  etc) 

With  question  five,  the  goal  was  to  find  the  number  of  hours  personnel  put  in 
directly  towards  the  DAB  process.  The  table  of  p-values  for  the  comparisons  of  the 
disciplines  below  show  that  in  all  cases  the  null  hypothesis  was  rejected.  On  the  question 
of  how  many  hours  are  spent  in  support  of  DAB  approval,  the  difference  in  means  in  the 
comparison  of  DoD  5000  to  Space  and  DoD  to  C3I  were  statistically  significant.  The 
mean  value  for  the  low  estimates  for  DoD  was  418  hours  and  the  mean  value  for  the  high 
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estimates  was  1,068  hours.  For  Spaee,  the  mean  for  the  low  estimates  was  about  57 
hours  and  the  high  was  almost  184  hours.  For  C31  the  mean  for  the  low  was  12.5  hours 
and  the  high  was  34.  In  no  ease  did  the  ranges  on  the  mean  values  for  eaeh  diseipline 
interseet. 


Table  5.6:  Question  Five-Comparison  of  p-Values 


Question  5  p-Values  (.05  significance  ievei) 


COMPARISON 

LOW 

AVG 

HIGH 

Space  vs  5000 

0.0071 

0.001 

0.0005 

C3I  vs  5000 

0.0039 

0.0003 

0.0002 

Nuii  Hypothesis:  There  is  no  significant  difference 

Faii  to  reject:  No  difference  (Higiighted  in  Grey)-P>=.05 

Reject:  There  is  a  significant  difference  (P<.05) 

5.8  Question  Six 

Estimate  how  many  people  are  normally  involved  with  the  preparation  process. 

Question  six  provided  low,  average,  and  high  estimates  for  the  number  of  people 
involved  in  the  proeess  to  prepare  for  DAB  approval.  The  number  of  personnel  involved 
in  the  preparation  proeess  ineluded  those  ereating  slides,  preparing  briefings,  and 
supporting  the  DAB.  In  this  instanee  the  table  of  p-values  below  shows  that  we  failed  to 
rejeet  the  null  hypothesis  in  all  eases.  The  mean  responses  from  the  populations  of  DoD 
5000,  Spaee  and  C31  were  not  statistieally  signifieantly  different.  The  only  eomparison 
that  eame  elose  to  rejeeting  the  null  was  eomparing  high  responses  from  DoD  5000  to 
C31. 
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Table  5.7;  Question  Six-Comparison  of  p-Values 


Question  6  p-Values  (.05  significance  ievei) 


COMPARISON 

LOW 

AVG 

HIGH 

Space  vs  5000 

0.2225 

0.2213 

0.1415 

031  vs  5000 

0.1906 

0.1331 

0.0587 

Nuii  Hypothesis:  There  is  no  significant  difference 

Faii  to  reject:  No  difference  (Higiighted  in  Grey)-P>=.05 

Reject:  There  is  a  significant  difference  (P<.05) 

5.9  Question  Seven 

Estimate  the  cost  per  hour  for  each  person  involved  in  the  process. 

Question  Seven  provides  the  last  portion  of  the  personnel  eost  estimate  for  the 
total  eost  of  oversight  by  providing  an  estimate  of  the  hourly  salary  of  eaeh  person 
involved  in  the  proeess.  The  table  of  the  p-values  for  the  comparisons  below  shows  that 
in  only  two  of  the  six  possible  cases  did  we  fail  to  reject  the  null  and  both  instances 
occurred  on  the  Space  to  DoD  5000  comparison.  The  two  cases  of  failing  to  reject 
occurred  when  comparing  low  estimates  for  cost  per  hour  and  high  estimates  for  cost  per 
hour.  The  comparison  of  average  responses  came  close  to  rejection  at  .07.  but  the 
comparison  of  low  values  was  very  high  at  almost  .62.  The  DoD  low  estimates  ranged 
from  a  cost  per  hour  per  person  of  $43  to  $60  with  a  mean  of  about  $48.  The  Space  low 
estimates  ranged  from  $32  to  $50  with  a  mean  of  almost  $46.  In  both  cases  the  mean 
from  one  discipline  fell  into  the  range  of  the  other. 
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Table  5.8:  Question  Seven-Comparison  of  p-Values 


Question  7  p-Values  (.05  significance  ievei) 


COMPARISON 

LOW 

AVG 

HIGH 

Space  vs  5000 

0.6153 

0.0746 

0.0018 

031  vs  5000 

0.0012 

0.0001 

0.0004 

Nuii  Hypothesis:  There  is  no  significant  difference 

Faii  to  reject:  No  difference  (Higiighted  in  Grey)-P>=.05 

Reject:  There  is  a  significant  difference  (P<.05) 

5.10  Personnel  Cost  Computation 

When  reviewing  the  total  estimates  provided  by  all  three  types  of  oversight  in  the 
table  below,  the  comparison  of  total  costs  for  personnel  show  large  value  differences. 

The  C3I  portion  is  significantly  smaller  than  the  NSSAP  and  DoDD  5000  oversight 
processes.  The  DoDD  5000  oversight  process  has  the  largest  range  for  all  three. 

Ranging  from  $120,000  to  over  $24M,  personnel  costs  create  a  large  portion  for  the  cost 
of  oversight  in  programs  under  the  centralized  control  of  DoDD  5000.  Space  is  estimated 
lower  at  $17,280  to  $2.4M,  but  doesn’t  compare  with  the  C3I  estimates  of  $4,200  to  just 
over  $144,000.  This  shows  that  using  the  C3I  approach  could  provide  some  potential 
cost  savings  in  the  oversight  process  by  placing  items  in  a  virtual  environment  and 
allowing  those  who  have  access  the  ability  to  view  at  their  leisure. 
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Table  5.9:  Personnel  Cost  Estimates  for  Eaeh  Oversight  Proeess 


5000  -  Personnel 

MEMBER 

Person-LOW 

Person-AVG 

Person-HIGH 

1 

$2,373,600.00 

$7,654,400.00 

$14,664,000.00 

2 

$2,250,000.00 

$7,312,500.00 

$22,440,000.00 

3 

$182,320.00 

$894,300.00 

$3,151,260.00 

4 

$120,000.00 

$1,200,000.00 

$7,200,000.00 

5 

$215,000.00 

$805,000.00 

$2,820,000.00 

MEAN 

$1,028,184.00 

$3,573,240.00 

$10,055,052.00 

STD  DEV 

$1,173,086.83 

$3,574,565.45 

$8,408,165.41 

Space  Personnel 

MEMBER 

Person-LOW 

Person-AVG 

Person-HIGH 

1 

$17,280.00 

$270,000.00 

$2,400,000.00 

2 

$40,000.00 

$252,000.00 

$1,200,000.00 

3 

$22,500.00 

$168,750.00 

$1,154,250.00 

4 

$19,800.00 

$315,000.00 

$1,750,000.00 

MEAN 

$24,895.00 

$251,437.50 

$1,626,062.50 

STD  DEV 

$10,293.11 

$61,161.80 

$582,659.43 

C3I  -  Personnel 

MEMBER 

Person-LOW 

Person-AVG 

Person-HIGH 

1 

$4,200.00 

$25,500.00 

$135,000.00 

2 

$7,200.00 

$25,200.00 

$129,600.00 

3 

$7,200.00 

$30,000.00 

$90,000.00 

4 

$7,680.00 

$33,600.00 

$144,000.00 

MEAN 

$6,570.00 

$28,575.00 

$124,650.00 

STD  DEV 

$1,596.12 

$4,005.31 

$23,851.42 

5.11  Question  Eight 

Estimate  how  many  meetings  are  normally  held  from  the  PEO  preparation, 
through  DAB  approval.  (This  includes  meetings  while  TDY or  TDY prep 
meetings). 

Question  eight  provides  insight  on  how  meetings  are  ineluded  into  the  oversight 
proeess  and  deals  speeifieally  with  the  number  of  meetings  that  are  held  during  one 
milestone  in  a  program.  The  table  of  p-values  shows  that  in  eomparing  the  means  from 
eaeh  population  our  results  were  not  statistieally  signifieant  to  disprove  the  null 
hypothesis.  There  were  two  instanees  where  we  eame  elose  to  rejeeting  the  null;  both 
eame  in  the  eomparison  of  DoD  5000  responses  to  Spaee  responses. 
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Table  5.10;  Question  Eight-Comparison  of  p-Values 


Question  8  p-Values  (.05  significance  ievei) 
COMPARISON 
Space  vs  5000 
031  vs  5000 


LOW  AVG  HIGH 


0.0667 

0.1117 

0.0768 

0.1129 

0.2234 

0.2686 

Nuii  Hypothesis:  There  is  no  significant  difference 

Faii  to  reject:  No  difference  (Higiighted  in  Grey)-P>=.05 

Reject:  There  is  a  significant  difference  (P<.05) 


5.12  Question  Nine 

What  do  you  estimate  as  the  length,  in  hours,  for  each  meeting? 

Question  nine  provided  an  estimate  for  the  length  of  eaeh  meeting  related  to  the 
DAB  milestone  approval  proeess.  The  table  of  p-values  on  the  eomparisons  show  that 
there  was  only  one  ease  where  the  null  hypothesis  was  rejeeted.  The  null  was  rejeeted 
when  eomparing  the  low  estimates  for  the  length  of  meetings  in  the  DoD  5000  proeess  to 
the  low  estimate  for  the  length  of  meetings  assoeiated  with  the  Spaee  proeess.  For  the 


DoD  proeess,  the  mean  duration  for  the  low  estimate  was  2.4  hours  and  just  under  an 
hour  for  the  spaee  proeess. 

Table  5.11;  Question  Nine-Comparison  of  p-Values 
Question  9  p-Vaiues  (.05  significance  ievei) 


COMPARISON 

LOW 

AVG 

HIGH 

Space  vs  5000 

0.0346 

0.3451 

0.4016 

C3I  vs  5000 

0.2977 

0.3489 

0.3326 

Nuii  Hypothesis:  There  is  no  significant  difference 

Faii  to  reject:  No  difference  (Higiighted  in  Grey)-P>=.05 

Reject:  There  is  a  significant  difference  (P<.05) 

5.13  Question  Ten 

What  is  the  cost  per  hour  of  each  person  involved  in  the  meetings? 
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Question  ten  provided  the  eost  per  person  for  meetings  conducted  in  support  of  a 


milestone  decision.  The  table  of  p-values  shows  that  comparison  between  space 
estimates  for  the  per  person  costs  and  DoD  5000  estimates  provided  the  only  instances 
where  we  failed  to  reject  the  null  hypothesis.  The  comparison  of  low  estimates  for  the 
cost  per  hour  for  an  individual  in  the  DoD  process  ranged  from  $43  to  $60  with  a  mean  of 
a  little  of  $50  an  for  space  ranged  from  $32  to  $50  with  a  mean  of  almost  $42.  The 
average  estimates  from  the  DoD  process  ranged  from  $46  to  $80  with  a  mean  of  just  over 
$60  and  the  estimates  from  the  space  process  ranged  from  $60  to  $75  with  a  mean  of  just 
over  $7 1 . 


Table  5.12:  Question  Ten-Comparison  of  p-Values 


Question  10  p-Values  (.05  significance  ievei) 


COMPARISON 

LOW 

AVG 

HIGH 

Space  vs  5000 

0.1722 

0.2536 

0.0049 

C3I  vs  5000 

0.0026 

0.0001 

0.0002 

Nuii  Hypothesis:  There  is  no  significant  difference 

Faii  to  reject:  No  difference  (Higiighted  in  Grey)-P>=.05 

Reject:  There  is  a  significant  difference  (P<.05) 


5.14  Meeting  Cost  Computation 

When  comparing  the  actual  cost  estimate  calculations,  an  interesting  point  came 
to  light.  As  seen  in  the  table  below,  the  total  cost  for  meetings  was  actually  lowest  in  the 
DoDD  5000  oversight  process.  More  information  would  be  required  to  make  any  large 
assumptions  as  to  why  this  had  occurred,  but  the  assumption  that  could  be  made  is  due  to 
the  lower  cost  per  hour  of  personnel  contributing  to  the  meeting  process.  The  other  areas 
for  the  cost  computation  come  in  significantly  higher  in  the  DoDD  portion  when 
comparing  to  the  other  oversight  processes.  Because  of  this  one  factor,  the  C3I  process 
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still  looks  as  though  it  could  statistically  come  in  with  a  lower  eost  for  meetings  than  the 
DoDD  5000  oversight  proeess.  The  other  interesting  oeeurrenee  is  the  high  eosts  that 
were  ealeulated  in  the  C3I  proeess.  They  were  assoeiated  with  one  high  point  that  may 
have  been  just  an  anomaly,  but  there  isn’t  suffieient  evidenee  to  keep  this  data  point  out. 


Table  5.13:  Meeting  Cost  Estimates  for  Eaeh  Oversight  Proeess 


5000  -  Meeting 

MEMBER 

Meeting-LOW 

Meeting-AVG 

Meeting-HiGH 

1 

$51,600.00 

$397,440.00 

$3,384,000.00 

2 

$34,560.00 

$270,000.00 

$2,736,000.00 

3 

$43,756.80 

$260,160.00 

$2,075,220.00 

4 

$7,200.00 

$128,000.00 

$1,800,000.00 

5 

$15,480.00 

$88,320.00 

$406,080.00 

MEAN 

$30,519.36 

$228,784.00 

$2,080,260.00 

STD  DEV 

$18,747.83 

$123,506.69 

$1,119,283.55 

Space  -  Meeting 

MEMBER 

Meeting-LOW 

Meeting-AVG 

Meeting-HiGH 

1 

$448.00 

$120,000.00 

$5,625,000.00 

2 

$4,000.00 

$48,000.00 

$1,701,000.00 

3 

$2,520.00 

$105,000.00 

$2,812,500.00 

4 

$3,000.00 

$134,062.50 

$1,680,000.00 

MEAN 

$2,492.00 

$101,765.63 

$2,954,625.00 

STD  DEV 

$1,495.64 

$37,757.05 

$1,857,179.46 

C3i  -  Meeting 

MEMBER 

Meeting-LOW 

Meeting-AVG 

Meeting-HiGH 

1 

$33,600.00 

$825,000.00 

$14,400,000.00 

2 

$18,000.00 

$112,500.00 

$448,000.00 

3 

$37,500.00 

$300,000.00 

$2,208,000.00 

4 

$39,600.00 

$257,040.00 

$3,686,400.00 

MEAN 

$32,175.00 

$373,635.00 

$5,185,600.00 

STD  DEV 

$9,771.51 

$311,415.24 

$6,283,940.47 

5.15  Total  Program  Cost  Comparison 

Reeall  from  ehapter  four  that  a  simple  algorithm  was  used  to  develop  estimates 
for  the  three  major  oversight  eost  portions  of  a  milestone  review  for  an  “inside  the  box” 
MDAP  and  then  those  individual  estimates  were  added  together  to  arrive  at  an  overall 
estimate  for  the  cost  of  one  milestone  deeision  point.  A  total  program  eost  by  was  then 
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estimated  by  multiplying  the  eost  of  one  milestone  by  three  sinee  there  are  three 
milestones  in  a  program’s  aequisition  lifeeyele.  This  algorithm  was  applied  to  the  data 
from  the  DoD  5000,  Spaee,  and  C3I  proeess  to  arrive  at  estimates  of  total  program  eost 
for  all  three  of  the  diseiplines.  This  portion  of  the  thesis  will  answer  the  researeh 
question,  “how  do  the  oversight  eosts  assoeiated  with  the  “box”  proeess  eompare  to  the 
oversight  eosts  of  the  other  proeesses?” 

Acquisition  reform  over  the  years  has  been  aimed  at  making  DoD  acquisition 
more  efficient  and  cost  effective.  Throughout  this  reform,  the  DoD  5000  acquisition 
process  is  assumed  to  be  the  primitive  way  of  doing  business,  so  the  pressing  question 
becomes  are  the  new  reforms  associated  with  the  space  and  C3I  acquisition  processes 
improving  those  acquisition  systems?  Below,  is  the  table  that  shows  the  estimates  that 
were  developed  for  programs  associated  with  the  box,  outside  the  box,  and  virtual  box 
processes. 


91 


Table  14;  Estimates  of  Total  Program  Cost  for  DoD  5000,  Spaee,  and  C3I 


5000  -  Program 

MEMBER 

Program  Low 

Program  Avg 

Program  High 

1 

$8,283,600.00 

$27,611,520.00 

$65,664,000.00 

2 

$8,581,680.00 

$27,931,500.00 

$94,968,000.00 

3 

$764,630.40 

$4,039,380.00 

$17,839,440.00 

4 

$417,600.00 

$4,284,000.00 

$30,750,000.00 

5 

$745,440.00 

$3,219,960.00 

$11,838,240.00 

MEAN 

$3,758,590.08 

$13,417,272.00 

$44,211,936.00 

STD  DEV 

$4,270,333.53 

$13,109,978.82 

$35,217,742.48 

Space  -  Program 

MEMBER 

Program  Low 

Program  Avg 

Program  High 

1 

$82,434.00 

$1,602,000.00 

$26,019,000.00 

2 

$222,000.00 

$1,485,000.00 

$11,443,500.00 

3 

$107,685.00 

$1,271,250.00 

$13,790,250.00 

4 

$173,400.00 

$1,779,187.50 

$12,666,000.00 

MEAN 

$146,379.75 

$1,534,359.38 

$15,979,687.50 

STD  DEV 

$63,337.35 

$213,056.30 

$6,761,137.82 

C3I  -  Program 

MEMBER 

Program  Low 

Program  Avg 

Program  High 

1 

$221,400.00 

$3,751,500.00 

$52,605,000.00 

2 

$201,600.00 

$696,600.00 

$3,028,800.00 

3 

$314,100.00 

$2,190,000.00 

$13,194,000.00 

4 

$861,840.00 

$4,471,920.00 

$25,891,200.00 

MEAN 

$399,735.00 

$2,777,505.00 

$23,679,750.00 

STD  DEV 

$311,948.30 

$1,682,759.26 

$21,431,855.48 

The  tables  provide  low,  average,  and  high  estimates  of  program  eosts  for  eaeh  of  the 
proeesses.  The  tables  also  break  out  those  estimates  by  Delphi  expert  panel  member,  but 
the  foeus  of  this  analysis  will  be  on  the  means  for  eaeh  proeess.  As  one  ean  see  from  the 
data,  the  estimates  for  the  total  program  eost  of  those  programs  going  through  the  DoD 
5000  proeess  are  the  highest  ranging  from  a  low  estimate  of  almost  $3.8  million  to  a  high 
estimate  of  over  $44  million.  It  was  shown  in  ehapter  4  that  the  program  eost  estimate 
arrived  at  in  this  thesis  coineides  with  estimates  developed  in  a  previous  study  of  the  eost 
of  oversight  of  DoD  5000  aequisition.  The  expeetation  prior  to  eondueting  this  thesis 
was  that  assuming  the  DoD  5000  proeess  is  the  baseline  proeess  targeted  to  be  reformed. 
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then  it  should  have  the  highest  oversight  eosts  assoeiated  with  it  and  the  results  of  this 
thesis  supported  that  expeetation.  The  C3I  proeess  was  seeond  to  the  DoD  proeess  in 
total  program  eost  with  a  low  estimate  of  almost  $400  thousand  ranging  to  a  high 
estimate  of  almost  $24  million.  The  C3I  proeess  was  nieknamed  “virtual  box”  early  on  in 
this  researeh  due  its  adherence  to  the  procedures  outlined  in  the  DoD  5000  mixed  with 
innovations  in  technology  aimed  at  reducing  oversight  costs.  Early  expectations  were 
that  the  innovations  associated  with  the  C3I  process  would  reduce  oversight  costs  below 
that  of  the  standard  box  process  and  again  the  data  supported  that  expectation.  Finally, 
space  acquisition  came  in  with  the  lowest  oversight  costs  which  ranged  from  a  low  of  just 
over  $146  thousand  to  a  high  of  almost  $16  million.  The  space  acquisition  process  was 
dubbed  “outside  the  box”  since  space  acquisition  operates  in  an  environment  that  steps 
outside  the  framework  of  rules  and  procedures  outlined  in  the  DoD  5000  series.  Early 
expectations  were  that  space  would  have  the  lowest  oversight  costs  because  space 
acquisition  had  the  freedom  to  develop  acquisition  rules  and  procedures  that  best  suited 
acquisition  in  the  space  world.  Again  the  expectation  was  met.  The  next  step  for  the 
comparison  of  total  program  costs  by  oversight  discipline  was  to  enter  the  data  in  JMP 
5.0.1  and  perform  statistical  analysis  and  hypothesis  testing  similar  to  that  conducted  in 
questions  2-10.  The  table  below  displays  the  resulting  p-values. 


Table  5.15:  p-Values  for  Comparison  of  Total  Program  Costs 


Total  Cost 

p-Values  (.05  significance  level) 

COMPARISON 

LOW 

AVG 

HIGH 

Space  vs  C3I 

0.1625 

0.1932 

0.5188 

Space  vs  5000 

0.1393 

0.117 

0.1629 

5000  vs  031 

0.1655 

0.1557 

0.343 
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The  chart  quickly  shows  that  in  no  case  were  the  p-values  high  enough  to  reject  the  null 
hypothesis  that  there  is  no  statistical  difference  between  the  mean  total  program  costs  of 
the  populations  of  DoDD  5000  oversight,  space  oversight  or  C3I  oversight.  The  p-values 
in  table  5.15  were  quite  surprising  considering  the  range  of  cost  differences  on  total 
program  cost.  These  ranges  for  each  process  can  be  seen  in  Table  5.16.  Section  5.17  will 
show  however,  that  each  oversight  process  has  its  own  cost  savings  potential. 


Table  5.16:  Total  Cost  Ranges  by  Oversight  Process  within  Range 


Process 

Low  Range 

Avg  Range 

High  Range 

NSSAP 

$82,434  to  $222,000 

$1,271,000  to  $1,779,188 

$11,443,500  to  $26,019,000 

C3I 

$201,000  to  $861,840 

$696,600  to  $4,471,920 

$3,028,800  to  $52,605,000 

5000 

$417,600  to  $8,581,680 

$3,21 9,960  to  $27,931 ,500 

$1 1 ,838,240  to  $94,968,000 

5.16  Summary  of  Results 

This  research  effort  was  aimed  at  answering  three  research  questions: 

1)  What  is  the  cost  of  oversight  of  programs  under  the  “box”  process? 

2)  How  do  oversight  costs  for  box  programs  compare  to  oversight  costs  of 
programs  under  space  and  C3I  processes? 

3)  What  are  the  key  oversight  cost  drivers? 

Questions  one  and  three  were  answered  in  chapter  4  by  using  Delphi  survey 
techniques  on  a  five  person  expert  panel.  Question  two  was  answered  in  chapter  five  by 
comparing  the  total  program  costs  developed  in  chapter  4  for  the  box  process  to  those 
program  cost  estimates  developed  for  space  and  C3I  in  other  research  processes  using  the 
same  methodology.  We  found  that  the  actual  oversight  costs  matched  up  with  our 
expected  oversight  costs  for  programs  under  each  discipline.  Two  questions  now  come  to 
mind.  The  first  question  is  if  space  acquisition  costs  less  to  oversee,  what  is  being  done 
there  that  can  be  done  in  DoD  5000  acquisition?  C3I  acquisition  remains  within  the  DoD 
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5000  framework  yet  had  lower  oversight  eosts  in  large  part  due  to  their  reduction  in 
number  of  people  needing  to  go  TDY.  The  second  question  then  is  will  some  of  the 
virtual  box  processes  employed  in  the  C3I  world  work  for  DoD  5000  acquisition?  There 
are  no  easy  answers  or  DoD  acquisition  would  be  perfect  with  no  need  for  improvement. 
The  best  answer  may  be  to  consider  the  possibility  of  a  hybrid  mix  of  the  most  effective 
procedures  from  each  acquisition  process. 

5.17  Recommendations 

To  enhance  the  oversight  process  and  decrease  the  potential  cost  of  oversight  that 
all  three  oversight  processes  posses,  it  is  then  our  recommendation  that  the  process  of 
IPA  and  C3I  be  merged  into  an  oversight  process  that  allows  approval  by  an  independent 
board,  but  information  for  those  meetings  and  approval  processes  should  be  available 
over  a  virtual  process.  This  hybrid  oversight  could  potentially  reduce  the  oversight  cost 
ranges  that  were  listed  above  in  Table  5.16  to  the  smaller  ranges  which  are  located  in 
Table  5.17.  By  combining  both  processes,  the  total  for  potential  savings  ranges  from 
$40,000  to  a  little  over  $74M  per  program.  These  funds  could  be  realized  either  directly 
or  indirectly,  either  in  saved  man-hours  or  actual  bottom-line  budget  savings.  Either  way, 
this  move  has  a  potential  for  significant  savings  to  the  point  that  some  sort  of  live 
program  test  or  feasibility  study  should  be  performed. 


Table  5.17:  Proposed  Combined  Oversight  Process  IPA/C3I 


Process 

Low  Range 

Avg  Range 

high  Range 

Proposal 

$43, 194  to  $140,040 

$652,500  to  $1,087,988 

$7,200,000  to  $20,046,000 
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5.18  Follow-on  Possibilities 


There  is  a  potential  for  further  research  in  this  area.  An  analysis  gathering  more 
panel  members  and  the  inclusion  of  the  Ballistic  Missile  Defense  Agency  is  a  possibility 
that  could  add  even  more  possibilities  of  cost  savings  for  the  Department  of  Defense  and 
the  oversight  of  future  acquisition  processes.  It  is  also  important  to  note  that  these 
experts,  from  all  three  panels,  were  mostly  from  Air  Force  sources  or  backgrounds.  An 
excellent  follow-on  would  gather  data  from  other  services  and  compare  the  data  to  this 
study.  Another  option  is  to  gather  additional  data  and  utilize  simulation  to  increase  the 
number  of  data  points  collected.  By  adding  the  additional  data  points,  a  more  accurate 
range  of  estimates  could  be  developed.  Another  possibility  is  to  see  if  the  costs  are 
increased  or  decreased  as  programs  are  delayed  in  the  process.  This  research  only 
scratches  the  surface  on  the  potential  research  trying  to  capture  the  cost  of  oversight. 
Funds  expended  in  the  oversight  process  aren’t  always  budgeted  dollars,  but  they  do  cost 
the  government  in  direct  or  indirect  costs.  Overall,  this  research  provided  the  basis  for 
the  identification  for  potential  cost  savings  in  the  acquisition  environment. 
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